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L Overview of First Year Accomplishments 

This one-year study was conducted to determine what students were doing as they 
engaged in the Chelsea Bank simulation activities. The primary data source was a set of 22 hours 
of videotaped classroom activities. In addition, classroom observations; teacher, student, and 
administrator interviews; and reviews of classroom assignments and projects were used in the 
study. 

The major goal of this analysis was to build a knowledge base for the development of 
assessments that would guide instruction and focus teachers and students as they used the 
simulation materials. In addition to the development of assessments, the research was designed to 
imderstand how the program is functioning in order to make it more effective, particularly in the 
area of teacher education. 

A brief overview of the major outcomes and the activities for the first year summarize the 
status of the study. 

What major outcomes were achieved? 

The project planned to achieve five major outcomes during the first year. These outcomes 
were originally listed in the research proposal and are repeated here so that the reader does not 
have to look back at the proposal. After each of the five outcomes is a brief statement about what 
was accomplished regarding the outcome. The complete achievement for each of these outcomes 
is listed in the body of the report. 



1. Analysis of all student/teacher activities as these relate to the SCANS competencies. 

Additional categories will be added to the SCANS competencies as needed. The analysis will 
include a discussion of areas of the SCANS competencies which are emphasized and those 
which are not emphasized. 

Status: This analysis has been completed. The results which are included in Section III of the 
report indicate which categories of the SCANS competencies are emphasized in the Chelsea 
Bank program. 



2. Outline of the structure/format for performance assessments that could be developed to assess 
program outcomes. 

Status: An outline for a comprehensive assessment plan has been developed. This plan has been 
included in the proposal for funding for years two and three. In addition, the results that gave rise 
to the plan are described in Section IV of the report. The assessments will cover the needs of 
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teachers, students, and those who make decisions about program adoption. Assessments will 
include both ongoing process assessments and end-of-program product assessments. Specific 
plans for individual assessments have been developed. 



3. Analysis of the program impact on the school and classroom curriculum. 



Status: Data were 
gathered 

throughout the year. Included were interviews with teachers, administrators, and students. 
Teacher projects and assignments were also collected. The results of the analysis of the 
material’s impact on schools have been disappointing. This does not mean that the program has 
not had a significant impact on schools. It does mean that we are not certain of the nature of the 
impact or of how significant the impact has been. We feel that the classroom curriculum impact 
question will be answered much more directly after various assessments have been developed 
and implemented. In addition, the classroom case studies conducted during the second semester 
(See Section V) provide results regarding this outcome. 



4. A report on the problem-solving competencies emphasized by the program. 



Status: This 
outcome has 



been completed. A quantitative/qualitative analysis of the videotapes, teacher and student 
interviews, and various Chelsea Bank materials has been completed and is reported in Section IV 
of the report. Much of this information, along with the SCANS analyses will focus the design 
and objectives of the assessments. 



5. An initial analysis of the program components and their impact on learning outcomes. This 
will be presented in a set of hypotheses as to which aspects of the program are producing 
which results. 

Status: The results of a careful review of the scenarios has been completed and is reported in 
Section II of the research report. This analysis was completed early in the year and provides an 
excellent context to study the videotapes. Many of the teacher and student behaviors as they 
engaged in the simulations are better imderstood in light of the structure and organization of the 
scenarios. The analysis of the program components will have a significant impact on the 
assessments developed in the next two years. 
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What major activities were completed? 

The research project called for three major activities. A fourth activity was added after the 
research was begun. These are described below along with a brief statement about the status of 
each. 



1 . The documentation of evidence of student activity in SCANS categories while the students 
are engaged in using the Chelsea Bank simulation materials. The documentation will also 
focus on the problem-solving, cognitive, and metacognitive strategies in which students 
engage. The structure of the categories will emerge as the analysis takes place. However, the 
SCANS report competencies will serve as the initial guiding structure. Data will be gathered 
from a) 22 hours of existing videotapes, b) logs of previous observations, c) classroom 
observations during the coming year, d) interviews with instructors about preparatory work, 
debriefing activities, and how modules integrate with other class activities. 

Status: This activity took most of the time and resources to complete. It involved a careful 
analysis of the tapes to see what existed, the development of various coding schemes, and the 
analysis and interpretation of results. The classroom observations, interviews, and teacher 
assignments and projects were used to expand the data base and to determine the validity of what 
was observed on the tapes. 



2. Gathering of samples of student work throughout the semester and use of this work as a 
partial basis for developing monitoring guidelines for assessing student growth in SCANS 
areas. If the student material is of sufficient quality, the guidelines will be used during the 
first phase to make judgments about the relative degree of student growth in the various 
SCANS areas. 

Status: This activity began slowly because teachers did not submit much material. That was to 
be expected since submitting the materials was at the teachers’ discretion and many were unsure 
about exactly what they should send. In addition, they were busy learning the procedures for 
implementing the Chelsea Bank simulation program in their classes. Some very useful materials 
were collected when the teachers were interviewed in their schools. Many possibilities for 
connections of the Chelsea Bank learning objectives to other aspects of the curriculum could be 
seen in these materials. There was no possibility to use this material as an indication of student 
growth in the various SCANS areas. 
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3. Gathering samples of effective lessons, assignments, and projects linked to computer 
modules. Through interviews with teachers, gather evidence and examples of factors 
supporting and hindering the development of such effective lessons. This information will, 
during the second phase, serve as a partial basis for developing research-based guidelines for 
integrating modules into existing curricula. 

Status: This activity was combined with the analysis of the simulation lessons. The artifacts 
were used to attempt to imderstand what teachers saw as important outcomes of the Chelsea 
Bank program. That was coupled with the analysis of the scenarios themselves and the teacher 
and student interviews. These three data sources became the focus of the major analysis of the 
lessons. The analysis suggested that the importance of teachers in the whole process could not be 
underestimated. The result of this analysis was a project for the second and third year which 
would focus on what teachers observe in learning and how they use those observations to further 
direct learning. 



4. It became obvious to the research team that the tapes were not adequate as a source to 
observe what students are learning over time. While the tapes do cover an entire semester, 
they do not systematically gather pertinent data. Therefore, the research team developed an 
additional major activity that was not a part of the original proposal. The activity was to 
conduct four focused case studies during the 1995-96 academic year. No additional budget 
was requested for this major activity. However, the Classroom, Inc. staff agreed to assist with 
it. Without their cooperation and assistance and the cooperation of the site teachers, this 
longitudinal analysis would not have been possible. 

The case studies included videotaping of three scenarios at the beginning, middle, and 
end of the Chelsea program; classroom observations; interviews with teachers and students; 
and the collection of various artifacts. These case studies were used in the analysis of changes in 
student behaviors that are reported in Section V. In addition, the case studies provided a rich 
source of information to examine differences and similarities between the students in the 
MindWorks tapes and those using the program in other schools. Finally, the case studies allowed 
the researchers to try out data collection procedures that were planned for use in more extensive 
studies during years two and three. 



O 

ERIC 

BSfim 



4 



7 



Chelsea Bank Simulation: Analysis of the Lessons 



II. Analysis of the Chelsea Bank Program Design 

In this section we examine the relationship between the design of the Chelsea Bank 
instructional materials and the learning goals that Chelsea is intended to support. Our interest 
includes the content, the nature of the learning activity, and the support for learning. That is, our 
goal was to examine the quality of the learning environment in relation to the learning goals. 
What is the background and context for the analysis? 

1 Simulations, in general, provide environments very supportive of a learner-centered 

approach to instmction. There are several features common to simulations that are relevant to 
this learner centeredness. First, the learner is faced with problems representative of real-world 
problems, e.g., landing a plane for a pilot or cashing a customer's check for a bank teller. Second, 
the simulations provide an authentic context for solving the problem. By an "authentic" context, 
we mean one that has a high degree of cognitive and procedural fidelity with the real-world 
environment. This means that the learner has the same decision-making responsibilities that 
would be encountered in the real world and the resources for that decision making are also 
reflective of the real world. The third characteristic of simulations is that they are simplified in 
• some way — ^they are not the real world. It is critical that the simplification does not reduce the 

cognitive and procedural fidelity of the simulation as that fidelity relates to the learning goals. 

I The simplification should be of tasks or cognitive requirements that are irrelevant to the 

cognitive and procedural demands associated with the learning goals. 

In sum, simulation environments are not only learner-centered (the learner is in control); 
but they are also inquiry-based (the learner is seeking information to make a decision). In current 
' parlance, the environment, at least in theory, provides a cognitive apprenticeship for the learner 

in which specific learning activities are anchored in the larger decision-making requirement. 

Simulations are extremely popular for job training in industry and the military (Towne, 
1995). They are particularly useful in letting learners practice dangerous tasks (e.g., flying), 
infrequent tasks such as equipment repair (Lesgold, LaJoie, Bunzo, and Eggan, 1988), and tasks 
that may extend over months or years in the real world and hence require too much time (e.g., 
city planning). 

While there are many instances of simulations in the world of work, there are very few 
simulations of workplace or citizenship situations for use in schools. It is not at all clear why this 
is, since so many of the features of simulations are consistent with the current learner and 
problem-centered pedagogical focus. It may be that the simulation designers have focused on 
high-end, complex data models that require computer power not generally available in the 
schools. It may also be that schools are resistant to the label of job training; and it is certainly the 
case that simulations have been associated with job training. 
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The Classroom Inc. simulations, which include Chelsea Bank, are among the few 
workplace simulations for use in schools. The software is designed to run on low-end machines, 
and the instructional philosophy is not to train individuals for a job, but rather to use the job 
context to teach more general skills. 

Our goal is to examine the success of Chelsea Bank, from a design perspective. However, 
as noted previously, we must look beyond simply the fidelity of the workplace representation if 
we are to assess the learning environment. Workplace simulations, in representing job 
environments, simply allow the individual to engage in the job activities — ^to make decisions and 
to see the consequences of the decisions. While the execution of the decisions (trying to fly the 
plane) and the feedback on success or failure (crash or not) will lead to some learning, more 
support is needed to create an effective learning environment. In particular, learning support — 
e.g., cognitive coaching that is focused on understanding, and support for reflective abstraction 
of the experience (Schon, 1987; Senge, 1994) — are essential for effective learning. These support 
activities may reside in the simulation, or they may be provided by a coach outside of the system. 

Our thinking about simulation learning environments outlined above led to the 
development of five criteria for the evaluation of simulation software. We wanted to compare our 
criteria with those derived from other efforts to evaluate the design of simulations. However, in 
our search of the literature we found surprisingly little. Indeed, as Jonassen {in press) notes, there 
is very little guidance on how to evaluate any kind of instruction that is aimed at developing 
problem-solving and decision-making skills in a domain. Reigeluth and Schwartz (1989) provide 
a strategy for designing and evaluating simulations, but it is from an information-processing 
perspective that focuses on getting the content into the head. It simply does not attend to the 
centrality of the decision making (inquiry) in the learning process; nor does it consider the 
importance of learner ownership in the learning and decision-making process. 

In the next two sections we present the five design evaluation criteria we have developed 
and then describe the learning objectives specified for Chelsea Bank. Following that, we look at 
each design criteria in relation to the objectives for Chelsea Bank. Let us emphasize that we see 
Chelsea Bank as a learning environment that extends beyond the software. Thus we examined 
both the design of the software and the instructions and guidance given to teachers in terms of 
creating and supporting the full learning environment. Later sections of this report examine the 
actual implementation of Chelsea Bank in the classroom and how teachers have attempted to go 
beyond the software. 
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What were we looking for in the design? 

Five principles have guided our evaluation of the Chelsea Bank design: 

1. The problem is authentic. 

We are looking for an environment that presents the learner with the kinds of problems 
real workers in the domain encounter. This includes both the problem as presented and the 
obstacles or issues that must be resolved in coming to a decision on the problem. 

The most common design flaw related to this principle is to oversimplify the problem so 
that it no longer represents even qualitatively, the kind of thinking called for in the actual 
environment {e.g., Spiro, Feltovitch, and Coulson, 1992). 

2. All learning arises out of working on the problem. 

There are two key issues here. First the learner must have ownership of the decision and 
the decision-making process. Second, all of the learning must arise out of the learners efforts to 
formulate a decision. The problem is the driving force for the learners' activities. Those activities 
include everything involved in gathering and evaluating information relevant to decision making. 
It is in the context of these activities that the learner encounters the concepts and issues and the 
complex inter-relationships in the domain. In addition to this development of domain knowledge, 
learners also, and most importantly, learn how to solve problems in the domain and they learn 
how to learn in the domain. 

A frequent design problem related to this principle arises from pre-teaching of concepts 
that should be learned in the context of the simulation. This pre-teaching reflects a view that 
simulations are a vehicle for practicing what is learned rather than an environment for learning. 
Another frequent design problem rests in providing hints as part of the simulation rather than as a 
coaching tool to be used in response to a learner’s request for help. The hints, which are 
frequently provided to make the simulation more interesting, result in a change in the learning 
task from one of determining what information is needed and what are logical sources for the 
information, to one of finding and interpreting the designer's clues. That is, learning is no longer 
arising out of work on the authentic problem as presented but rather it is organized around the 
"training" problem. 

3. Scaffolding in the environment supports the learning goals. 

The value of the simulation environment in comparison to on-the-job experience is 
considerable: The learner can safely explore the environment without real damage {e.g., crashing 
a plane); time can be compressed, permitting the learner to efficiently complete a lengthy 
i activity; irrelevant activities or requirements {e.g., "busy work") can be removed permitting more 

focused time on the central issues; time pressure can be removed allowing more time for thinking 
and learning, etc. 

O 7 
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While this support is generally known as "scaffolding" of the task, we think Carroll's 
(1990) term "training wheels" is perhaps abetter characterization. The goal in providing "training 
wheels" is that the individual will develop the skills that permit the training wheels to be 
removed and the learner to perform as a skilled/knowledgeable practitioner. 

In considering this variable in design, we must remember that scaffolding or training 
wheels will remove learning opportunities. For example, if we remove time pressures, the learner 
may not learn strategies that are effective in a time-pressure situation. If we offer the learners 
decision options, we take away their requirement to formulate a decision. If we remove particular 
data relevant to the problem, we lose training the learner to problem solve in an environment 
with that level of complexity. Thus, it is essential to examine the various sorts of scaffolding in a 
simulation to determine in what ways they may impact the cognitive demands that are central to 
the learning goals. 

4. Coaching supports the learner in understanding and applying the critical concepts 
and procedures related to the learning goals. 

Coaching may be in the computer system or it may be provided by the teacher or peers. It 
should be available during the analysis and decision-making process as a learning tool, rather 
than as a feedback tool after a decision is made. Again, the focus is on understanding (by 
definition, simulations are aimed at using information in decision making). Hence coaching is 
not effective if it simply tells the learner what to do next or provides hints as to what to do next. 

5. Reflection supports the learning in abstracting, synthesizing, indexing, 
and extending the learning that occurred in the simulation. 

Too often we engage in a project, do a lot of new things, and then never take the time to 
reflect on what was learned. As a consequence, we walk away with an experience but with little 
learning. It is absolutely essential that learners engage in reflective activity in which they can 
abstract or index the experience for future use. That is, the learner should summarize what is 
learned, identify keys to success, consider where else this new understanding might be applied, 
etc. 

What learning objectives does Chelsea Bank set out to achieve? 

A large and diverse set of learning objectives are identified in the Chelsea Bank 
materials. In the front matter of the Teachers Guide, fourteen learning objectives are identified. 

In addition to that listing, there are both general and specific objectives identified for each 
scenario. A sampling of the Chelsea 5an A: objectives is presented in Table 2.1. 

Clearly, the objectives, as stated, are at varying levels of specificity and have 
considerable overlap. Since learning arises out of the process of coming to a decision, we felt that 
all the learning objectives associated with the simulation could be organized under the decision- 
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making goals. Therefore, we attempted to organize these objectives by first identifying the 
decision-making requirements. The Chelsea Bank designers identify three areas of decision 
making that can serve as organizers for the learning objectives: banking practices, interpersonal 
skills, and ethical practices. 

Within each of these decision categories we classify objectives at three levels: 
imderstanding the concepts associated with the decision; imderstanding strategies associated with 
making a decision; and having the skills necessary to gather and analyze the data needed to make 
a decision. This matrix of decision category by class of learning objective is shown in Table 2.2. 
If we examine the objectives identified by the designers of Chelsea Bank in Table 2.1, most of 
the objectives are at the skill level and, for the most part, they are part of making decisions about 
banking practices. A more detailed consideration of the matrix follows. 



Table 2.1 Learning Objectives for Scenarios in Chelsea Bank Teachers Guide 


General skills and knowledge identified 


In the front matter 


For individual scenarios 


• identify relevant information 


• banking skills 


• apply relevant information 


• judgment 


• learn and apply rules and procedures 


• computation skills 


• setting priorities 


• ethics 


• apply math 


• interpersonal skills 


• read for detail 


• decision-making skills 


• read for comprehension 


• attention to detail 


• write at a business level 


• comprehension 


• set aside personal concerns 


• analytical skills 


• cooperate 


• customer relation skills 


• imderstand banking as a business 


• problem solving 


• imderstand banking services 




• understand credit 





Numerous objectives relate to banking practices. 

Understanding banking practices involves understanding the general purpose of banking, 
i.e., why people go to banks. It also includes understanding the purpose, from both a customer 
and a bank perspective, of the variety of transactions represented in the scenarios: check cashing, 
loans, credit card, deposits, withdrawals. The learners should be able to talk about how they 
would or could use a bank and they should be able to interpret someone else's description of their 
banking activity and ask questions about it. 
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Table 2.1 continued 
Detailed banking objectives 



• decide how to apply bank policy 

• detect problems that make checks not 
negotiable 

• compute money deposits 

• managing an unpleasant customer 

• managing an error (ethics) 

• detecting problems with checks 

• judgment about potential crime 

• dealing with bribery 

• put aside personal feelings 

• compare information sources 

• judge whether a person should have an 
account 

• calculate income and obligations 



verify customer’s identity 
determine negotiability of checks 
accepting deposits 
changing deposit slips 
making cash deposits 
verify identity of customer 
opening a new checking account 

conduct an interview to gather 
information 

handling customer complaints 
taking a loan application 
check credit history 



Table 2.2 Strategy for Classifying Learning Objectives 


Category of 
Learning Objectives 


Decisions 


Banking 

Practices 


Interpersonal 


Ethics 


Understand decision 
concepts 








Understand strategy for 
decision making 








Skills and knowledge for 
gathering and evaluating 
information 









As a bank teller, the learner faces a new "problem" with each customer. While the 
problems are related — part of a family of problems — each one is different in some way. Thus the 
learner must develop a general strategy for solving the banking problems. Each problem involves 
managing information: analyzing the information presented as part of the transaction request and 
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finding and interpreting the banking policy and procedures regarding that request. Hence, what is 
required is an information-analysis, problem-solving strategy. 

If the learners were training to become bank tellers, then the strategy might be a strict 
procedure very focused on the resources in the bank. However, the goal of the instruction is to 
develop more transferable problem-solving skills and thus we would want to support the learning 
and application of a more general strategy, e.g., the IDEAL model developed by Bransford and 
Stein (1984). 

In applying the problem-solving strategy, learners must have the necessary skills and 
knowledge needed to work with these particular resources. For example, they must be able to 
attend to detail, comprehend text, write, compute a deposit, compare information resources, etc. 

A significant number of the Classroom Inc. objectives fall within this category (see Table 2. 1). 
Interpersonal skiUs assure a satisfied customer. 

In order to develop interpersonal skills appropriate to the situation, learners must 
imderstand the concept of customer satisfaction. They must see that a happy customer is one who 
will continue to do business with the bank — a concern in the service industry. They must also 
imderstand that their efforts to satisfy the customer must occur within the constraints of the bank 
policy. 

Simply understanding the need for customer satisfaction is not enough. Rather, what is 
needed is a framework for achieving customer satisfaction. This framework might come from the 
study of interpersonal relations, sales strategies, conflict management, attitude formation and 
change, or a variety of self-help books. In essence, the learner must have a general framework for 
thinking about types of customer, types of contexts, and what it takes to achieve "satisfaction" 
with those customers in those contexts. 

Finally, learners must have the specific interpersonal skills for implementing a strategy. 
They must be able to "read" the customer’s reaction from both expressions and verbal statements. 
They must be able to say and act in the ways called for by the strategies that they understand. 

This skill execution is very much a situated skill, i.e., the learner must be able to respond to 
moment by moment changes in the customer attitude. 

Ethical practices involve recognizing what is fair, right, and within the rules. 

The learners need to develop a sense of right and wrong in terms of customers and the 
bank. They need to be able to interpret a situation and judge what approaches are fair and ethical 
in terms of treatment of the customer and in terms of their responsibility to the bank. The "rules" 
for ethical behavior are contained in the banking policy, e.g., the rules state what to do if the 
customer makes an error in calculating. However, we presume the concern for ethics in the 
learning goals for Chelsea Bank extends beyond the simply following of rules to the 
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development of a sense of respect for "doing the right thing." Thus, while the strategies and 
tactics for ethical behavior are contained in the banking rules, the broader sense of right and 
wrong is the focus of this learning objective. 

The need to cooperate is dictated by implementation, not design. 

In addition to the goals described above, there is a goal of learning how to work with 
others cooperatively, learning how to negotiate and share decision making. This goal arises from 
the directions to the teachers to have students work in cooperative groups. Since this is part of the 
implementation strategy, rather than the design strategy, we will not consider this goal in the 
design review. 

To what degree does the program’s design support its learning goals? 

We now seek to apply the design evaluation criteria to the Chelsea Bank simulation to 
determine to what degree the design supports the learning goals identified for the simulation. We 
will consider each criterion in turn. 

Is the problem authentic? 

Banking Practices. The problems, as presented to the learners, are representative of the 
range of banking tasks a teller would be expected to do. The service requests by the customers 
are typical (make a deposit, cash a check, handle a complaint, make a loan, open an account, or 
issue a credit card). Furthermore the types of problems and the frequency of occurrence of the 
types seems reasonable and typical.' The teller transactions have a wide variety of errors in the 
materials the customer submits and thus a detailed examination of the materials is required. 

There is also an occasional need to obtain sign-off from the customer service representative and 
an occasional need to check balances. The customer service representative opens accounts, issues 
loans, and resolves complaints. Thus, Chelsea Bank receives high marks on this criterion as it 
relates to the learning goal of banking practices. 

Interpersonal skills . In every problem the learner meets a customer with a particular 
personality representing the range of types of customers that might be encountered in a bank. 

Thus the presenting problem does address the interpersonal and ethical goals. However, the 
learner does not encounter the complexities and problems associated with attempting to satisfy a 
customer. There is only the presenting personality of the customer; there is no interaction with 
the customer, and hence there is no potential for interpersonal problems to arise. The learner 
simply responds to the customer following bank policy — with no interaction. The learners do not 



An exception to this is Scenario 7 in which a rock star conies to die window and wants to make a deposit. He asks 
for fast service and slips a $100 tip and an autographed picture into die- teller cage. We suspect die audienticity of 
this scenario is rather low outside of a 90210 context. 



O 12 

ERIC 

hiniiiHiffnffTiaiadi 



15 



Chelsea Bank Simulation: Analysis of the Lessons 



need to understand interpersonal skills nor do they need to have strategies of specific skills in 
conducting an interaction. 

It is important to note, however, that presenting customers with interesting personalities 
is an important strategy — ^perhaps the primary design strategy — for making the scenarios 
interesting and giving them an individual character. Thus, there may well be a strong 
motivational component to providing customers with personality. It is also possible for the 
teachers to organize activities outside of the scenario whether the students could use these 
personalities in the study of the concept of customer satisfaction and the strategies for pleasing 
different types of customers in classroom. However this sort of learning is not supported in the 
simulation itself^ 

Ethics . The learners encounter several situations that call for ethical decision making. 
However, the ethical considerations are not with the strong competing motives associated with 
real ethical problems (real emotions associated with, e.g., the need for money for my family and 
the opportunity to pocket part of a customer’s deposit without anyone knowing). Indeed, the 
emotional component of the ethical context is difficult to capture in any instructional 
environment. But beyond the emotional component, there is no dilemma — it is always simply a 
matter of following banking rules. Hence the problem solving is not in terms of ethics but in 
terms of rule following. As with interpersonal skills, the ethical contexts can provide a basis for 
instruction and discussion outside of the simulation — ^however, we would expect this learning to 
be very academic with little transfer to real life. 

Does all learning arise out of working on the problem? 

One of the key features of Chelsea Bank is the lack of instructions to the learners. 
Nowhere in a scenario are the learners told what to do next, what the functionality of a particular 
icon is, or how to use the tool represented by the icon.^ Thus, the learners are responsible for 
problem solving much as a teller would be on the job. They must decide (recall fi-om initial 
training, infer fi-om analyzing the problem) what needs to be done or where to go for information 
on what is to be done. When a step is taken, the learners must decide on the next step (or on 
whether there is a next step) and how to execute that step. And, as in the real world, if the 
learners are not sure what to do, they must initiate exploratory activity or ask for advice fi-om 
peers or the teacher. Some examples: 



^ We will discuss the Teachers Guide support for teaching interpersonal skills when we look at coaching and 
reflection. 

^ There are two exceptions. In customer service scenarios, where an application is being made, the application form 
on the desk is outlined in yellow. In customer service scenarios where there is a complaint, the telephone remains 
highlighted after it finished ringing. While the telephone highlighting may be necessary (unless the ringing 
continued until it was answered), it is unclear why the application form is highlighted. 
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• At the start of a scenario, a customer approaches, places some materials into the window, 
and states a transaction goal. That is all that happens. The next move is entirely up to the 
learners — ^who must come to recognize that they must take the initiative rather than 
waiting to be told. 

• The same application form is used for both new accounts and loans, however only some 
of the questions are appropriate for new accounts. The questions are not marked "for loan 
applications only," but rather the learners must learn what questions to ask. (This can be 
inferred by looking in the manual to see what types of information are required.) If the 
learner asks a question inappropriate to the transaction, the customer responds angrily. 

• It is up to the learners to determine when they are ready to make a decision, and then they 
must know to follow the path from the computer icon to the "decision" button and then to 
the decision options. Similarly after a decision is made, there is no system response. The 
learners must remember that tiiey are responsible for filing a report — ^the "report" button 
being one of three other buttons along with the "decision" button. 

Clearly, this is learner-centered instruction with learning situated in the learners' work on 
the problem. There are three exceptions to this, two in the pre-scenario activities presented in the 
Teachers Guide and the other in the memo given to the learner at the start of the scenario. In the 
pre-scenario activity, the teacher is directed to teach the students key vocabulary. This pre- 
teaching removes the responsibility from the learners of monitoring their comprehension of the 
vocabulary during their work on the scenario — one of the learning goals associated with carrying 
out the analyses of the banking transaction requests. 

The pre-instruction guidance also offers discussion questions for the teacher to present 
before the scenario. These discussion questions almost always focus on the interpersonal and the 
ethical issues in the problem. Thus the students can quickly learn to look for those issues when 
they begin work on the scenario. Their learning of problem finding skills is short circuited. 

Finally, when the learner begins a scenario, an office memo raises an issue that is relevant 
to solving the problem. As with the pre-scenario questions, the memos tend to address 
interpersonal and ethical issues and this points the students to the key issues in the scenario. 

The use of pre-scenario direct instruction reflects a common instructional concern — ^that 
the learners must be jump-started on the problem — ^that they cannot learn adequately in the 
context of the problem. However, as work with problem-based learning has demonstrated 
(Savery and Duffy, 1995), learning can successfully occur without pre-training. 

With that overview, we now consider the degree to which learning related to each 
learning objective is situated in the problem. 

Banking practices . Each scenario presents a different problem to the learner, and the basic 
task is to determine the procedure for completing the transaction. In almost all cases, there is 
little need to understand the banking concepts involved — it is only necessary to determine how to 
complete the transaction. The exceptions are the scenarios in which the learner must determine if 
the customer should receive a loan or a credit card. But even in these cases, the learner is only 
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taking the bank perspective — there is little need to understand the value of the card or loan to the 
customer. 

The fact that there are seven bank teller scenarios encourages the learner to develop a 
problem-solving strategy that will work across scenarios. However, as we will see in the 
discussion of coaching and reflecting, there is no guidance in developing that strategy or in 
reflecting on whatever strategy evolved over the course of the seven scenarios. The string of 
scenarios simply affords the opportunity for learners to develop a problem-solving strategy. 

The scenarios are excellent in demanding the skills related to solving the problem. The 
learners are in control of the problem-solving activity and therefore they must analyze the issues, 
decide what information is required, and gather that information. There are no additional hints 
and no feedback on whether the learner is doing it "right." This problem-solving activity requires 
skills of reading for comprehension and reading for detail, doing basic arithmetic, doing 
systematic comparisons, etc. In essence, the variety of skills identified by Classroom Inc. as 
goals for the instruction are called upon in the scenarios. 

Interpersonal skills . As noted previously, the pre-scenario discussion and the comment in 
the memo most often raise the interpersonal issues. Thus there is pre-teaching and focusing on 
the interpersonal skills and this learning is not situated in the scenario. Rather it reflects a 
traditional teaching environment. 

The scenario itself places little demand on the learner to understand the concept of 
customer satisfaction. Rather, through the feedback on the scenarios, students learn that the 
customer will be satisfied if they simply follow bank policy. Since the customer never reacts to 
the learner, there is no requirement to understand strategies for satisfying customers (beyond 
following the rules of banking). While the policy manual lists policy in dealing with customers, 
there is no real need to understand or use those principles. 

Finally, since there is no interaction with the customer, the learner cannot practice 
interpersonal skills. 

Ethics . There are only a few cases in the program involving ethics, and the considerations 
are virtually identical to those for interpersonal skills. The learner is not faced with the emotions 
associated with an ethical dilemma, and the simple response called for in each ethical case is to 
follow banking rules. Thus it is banking rules rather than ethics tliat guides behavior. 

Summary and recommendations . The lack of instructions on what to do next does 
empower the learner in the problem-solving process so that almost all learning is situated in the 
problem. The exceptions are in the pre-teaching that occurs prior to the scenario or in introducing 
the scenario: vocabulary training and hints and discussion questions related to interpersonal and 
ethical issues. 
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However, situating learning in the problem is but one part of the learning issue. We must 
also consider what learning is demanded. In reviewing the scenario, we find most of the learning 
requirements to be in the basic skills involved in evaluating the transaction request, i.e., in the 
skills related to banking practice. There is little demand for understanding banking practices, for 
understanding the concept of customer satisfaction, or for interpersonal strategies or skills . 

We recommend the introduction of richer problems, problems that extend beyond simply 
doing a transaction, in order to more fully engage the learner in the banking and customer 
satisfaction concepts and problem-solving strategies. For example, a scenario may call upon the 
learner to help the customer select banking options. We also recommend moving the pre-scenario 
activities to the post scenario or even using them as discussion topics during work on the 
scenario. Interspersing reflection and discussion during a scenario could lead to a much richer 
learning environment. For example, the teacher could pause and ask the students to think about 
what words they did not understand — or which were new words. The goal is to move the 
discussion to a point after the learners become aware of the relevance of the issue or can see the 
relevance of the issue through their work on the problem. Finally, the glossary/dictionary in the 
Policy Manual could be enriched so that the students would have a resource to check meaning 
when a question arose within the context of the problem. 

Does the scaffolding support the learning goals? 

In comparing the Chelsea Bank simulation to the real banking task, we note that the 
learners obviously operate in a computing environment rather than in the physical banking 
environment. Beyond this basic characteristic, we can identify four ways in which the tasks 
associated with the banking, interpersonal, and ethical decision making have been simplified or 
removed. After examining these four scaffolding strategies, we will examine the impact on 
achieving each of the learning goals. 

Simplified banking tools . A primary scaffolding is in the creation of simplified banking 
tools: credit reports, account summaries, a banking manual, and an application form. The credit 
reports and account data are reasonably rich in information and the procedures described in the 
manual, while task-relevant, are not tied directly to the tasks. Hence inferencing skills are 
required to apply the procedures. However, while the individual entries are rich, there are a 
limited number of entries in each resource. For example, there is no credit information on a 
person unless a credit check is required for the person. And the only person with a detailed credit 
history is a person having credit problems. Finally, there is no irrelevant information in the 
manual — each procedure in the manual is relevant to at least one scenario. 

The primary impact of this scaffolding is on learning related to banking practices. Most 
obviously, the complexity of the skills of reading for detail, following procedures, comparing 
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information, calculating, and judging the relevance of information are all reduced. Hence less 
learning of these skills is expected. Of course the banking tasks had to be simplified. Middle 
school students ought not be expected to work with the level of complexity that real bank 
persoimel work with. Thus this simplification makes it possible for the students to "enter" the 
world of banking. However, we question whether there might be too much simplification of the 
materials. It seems to us that middle school students would be able to work with this level of 
complexity quite easily. They could work on the problems without being adequately challenged 
in many of the basic skill and problem-solving areas listed as learning objectives. The only way 
to determine if this is so is through pretesting, but that pretesting must look at the specific skill 
demands, not just at the overall ability of the student to complete a scenario. That is, the goal is 
to fine-tune each of the banking resources so the reading, calculating, judgments, comparing for 
detail, and other basic skills are adequately challenged in using each resource. 

In addition to impacting skill development in the area of banking, we would expect this 
simplification of resources to also impact the learning of banking concepts. That is, we suspect 
that a more complex set of banking resources would result in the learners having to make more 
detailed inferences about the particular banking transaction and therefore in becoming more 
engaged in understanding the banking concepts. Finally, the simplification also reduces the need 
for a rich problem-solving heuristic — ^there is more focus on following procedures to collect data 
than on determining which information in a particular resource is relevant. Let us emphasize that 
some simplification of resources is necessary to permit the learner to enter the world of banking, 
but since these resources are directly involved in the banking practice, the simplification must 
also reduce the learning opportunity related to banking practices. Hence it is important to fine- 
tune the resources to ensure that the students are adequately challenged in regards to each 
relevant learning objective. 

We do not see this scaffolding as impacting on the objectives related to interpersonal 
skills or ethics. If our focus were simply interpersonal skills or ethics, we would want to simplify 
these materials as much as possible. 

Suspension of time . In the real world of banking, decisions must be made quickly and 
transactions processed smoothly. In the Chelsea Bank simulations there is no time factor; 
students can work on any aspect of a scenario for as long as they want (or the teacher will permit) 
with no adverse consequences. Removing the time pressure makes it possible for the students to 
analyze and evaluate the evidence and their problem-solving strategies. It also affords the time 
for the teacher to coach and pose reflective questions. In essence the removal of time pressure 
offers the opportunity to study the concepts, develop and evaluate problem-solving strategies, 
and carefully review and evaluate the data. 
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This is a scaffolding strategy that can benefit learning in all three areas of decision 
making. The students can take the time to evaluate the overall task they are involved in and 
develop a rich understanding of the banking, interpersonal, and ethical concepts that apply as 
well as develop the strategy for reaching a decision in each area. Since the students do not 
interact with the customer, the learning of the actual interpersonal skills is not impacted. It is 
only in relation to learning the concepts and strategies for customer satisfaction that greater 
learning opportunity is afforded. But, since the students are engaged in basic skills related to 
banking practices (comparing information, following procedures, etc), greater learning 
opportunity for developing these skills is afforded by the suspension of time. 

It is important to note that we described this scaffolding strategy as simply "affording" 
the opportimity for more learning. Whether deeper learning occurs depends on the task demands 
(as determined by the other design features of the simulation, e.g., the complexity of the 
resources), the coaching, and the reflective questions. 

Final decision ovtions . The learners are left in control of their problem solving until a 
final decision is required; they can review any data in any order up until that point. However, 
when it is time to make a decision as to how to respond to the customer, they are presented with 
3 to 5 decision options. The decision options identify the decision on the banking transaction, the 
treatment of the customer, and the ethical decision when there is one to be made. What are 
learners to do if they do not agree with any of the options? The message is, "do all you want on 
your own, but the decision is not really yours — we will set the parameters for the decision." 

As we worked through the scenarios, we found ourselves increasingly going to the 
decision options early to see the relevant parameters (things mentioned in the decision options) 
for the problem. Then, forearmed with knowledge of what we can decide and what is important, 
we returned to examine the customer materials. This was especially true in customer service, 
where there are no firm procedures and the number of ways of responding to the situation are 
wide open. For example, in one loan application scenario, a possible decision offered by the 
system is to award the loan only if the customer is willing to make a larger down payment on the 
car she wants to buy. This illustrates that there are many ways that a bank could make a loan. 
Thus learners should be able to think of reasonable options that go well beyond any limited 
number of alternatives the system might offer. 

As we discussed at the outset of this report, we view all learning to be situated in working 
on the problem — in the case of Chelsea Bank, this is coming to a decision on the banking 
transaction and how to treat the customer. Since the decision making is severely constrained by 
listing the decision options, this scaffolding strategy has a strong, and in our mind a negative, 
impact on all learning objectives. The learning requirements are reduced and what is still 
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required is situated in the decision options rather than in completing the transaction — an 
academic task rather than authentic problem solving. 

We strongly believe that this scaffolding should be eliminated — that students must be 
empowered to make and defend their own decisions. We understand that there is a right answer 
for the bank teller problems. However, there is no right answer for the interpersonal skills which 
go across all scenarios and there is no right answer for the customer service representative 
scenarios. Hence it is important to empower the students to develop and defend their own 
answers. 

We also recognize that teachers and students need feedback on the quality of their 
answers. However, this sort of feedback can easily be provided in a more authentic way by 
showing the students how other students have solved the problem and the reaction of banking 
representatives to these solutions. Thus we recommend that the decision options be eliminated 
from Chelsea Bank. We also recommend that each scenario be pretested with students and that a 
list of ten or more solutions be selected for use in Chelsea Bank as feedback to the students. Then 
when the scenario is completed, the students can compare their answers to those of other students 
and infer, based on the commentary associated with those other answers, what a bank supervisor 
or teller would tell them. This activity would provide a wonderful opportunity for the students to 
review the banking and customer satisfaction concepts and their problem-solving procedure. 

Executing the bankins transaction . The focus of the scenarios is on student involvement 
that begins with the entry of the customer to the point at which learners decide how to respond. 
The transaction is never carried out by the learners. It is as if the learner examines the case and 
then tells someone else what to do. 

The post decision activity would involve a dialogue with the customer, following 
procedures, and reading and calculating. Thus the biggest impact of this scaffolding strategy is to 
eliminate the learning of interpersonal skills — ^the learner does not have to actually interact with 
the customer. It also reduces the learning required regarding the customer satisfaction concepts 
and the development of interpersonal strategies. Since the learner does not have to respond to the 
customer, there is no need to get very involved in thinking through the concepts or strategies 
related to customer satisfaction. 

In contrast to the interpersonal skill area, this scaffolding increases the opportunities for 
learning in the area of banking practices. Understanding the banking concepts is situated 
primarily in coming to a decision related to the transaction rather than in the completion of the 
transaction. Similarly the richer problem-solving activity arises in that decision-making 
process — for the most part completing a transaction is by necessity a very specific procedural 
activity. Finally, while there will be basic skill and information management requirements in 
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completing the transaction, these skill requirements are at least equally represented in the process 
of coming to a decision. Hence there is little impact on these skills. 

In summary, this scaffolding strategy negatively impacts the learning associated with 
interpersonal skills but increases the learning related to some of the banking activities objectives. 
What coaching is provided for the learners? 

Fosnot (1989) describes coaching as asking learners questions on the cutting edge of their 
imderstanding. Thus the coach is there to challenge and direct the learners thinking on the 
problem. The coaching is of good learning practices — not of banking content. The teacher is 
there not to teach, but to support the development of effective problem solving and learning 
skills. Thus the coach should ask questions during the learning process about the meaning of 
terms, the imderstanding of the concepts, the problem-solving strategy being used, the current 
status of student thinking on the problem, etc. While the coach may "teach" by telling the 
students how to do something or the meaning of the term, this is not the focus of the coaching. It 
is not just helping the students get through the scenario but to aid the students in their learning 
activities associated with the objectives. 

There is no coaching support in the Chelsea Bank software. There is no help system built 
into Chelsea Bank to aid the student during the problem-solving activity. There is a requirement 
to identify the banking principle underlying the decision and to describe the impact of the 
decision on people, but that happens after decision making and is focused on S5mthesizing. 
Therefore we have classified it as reflective activity. 

An examination of the Teachers Guide for Chelsea Bank indicates that there is also no 
information for teachers on how to coach the students. The Guide offers advice to the teachers 
for what they should do before and after a scenario. However, there is no advice in the Teacher's 
Guide for what the teacher should do during the scenario. 

If the learning is challenging to the students, effective coaching is essential. While we 
think that Chelsea Bank affords many opportunities for learning, we thinking coaching is an 
essential factor in realizing those affordances. Indeed, coaching (and reflection) is essential to 
achieving the learning goals related to understanding the banking and interpersonal concepts as 
well as the problem-solving strategies in each of these areas. Since the scenario itself does not 
make many demands for understanding the concepts or problem-solving strategies these will 
only be learned through coaching and reflective activities. 

What kind of coaching support should be offered? We would argue that there are two 
types of coaching advice teachers need. First, there is general advice on how to coach; how to 
interact with the learners as they work. Should they approach the children with a question? 

Should they wait until called upon? How often should they visit a group and what role should 
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they play? What is the balance between visiting individual groups and talking with the whole 
class? How much can the work on the scenario be "interrupted"? How should the teacher deal 
with a student approach that is contrary to what the teacher thinks should be done — and even 
what the teacher guide suggests? What are "good" questions to be asking the students? Since 
this is a very specific learning environment with predetermined events unfolding, it offers a 
wonderful opportunity to provide concrete examples of very complex teaching concepts. 

Guidance on effective coaching strategies is missing not only from the scenario activity 
but also from the pre and post scenario suggestions. That is, while there are pre and post scenario 
activities to engage in, there is no overview of what should be accomplished before and after the 
scenario. For example, why are the vocabulary words identified pre-scenario? The design of the 
Chelsea Bank simulation and the other Classroom Inc. simulations reflect a very specific 
teaching philosophy — it would be useful to the teachers to understand how that philosophy 
might best inform their teaching practices. 

The second type of coaching advice is to offer specific advice relative to the learning 
objectives. We would hope that the coaching would involve developing a general model of 
problem solving then coaching the children to think about the model. The to-be-avoided 
coaching in this regard, is giving the students a specific procedure based on Chelsea Bank tools 
and design. That is, the goal is to learn to apply a problem-solving heuristic rather than learning a 
procedure for Chelsea. Similarly, guidance is needed on how to coach basic skill development, 
understanding bank concepts, and interpreting situations from multiple perspectives 
(interpersonal skills). 

How is reflection supported? 

Two types of reflective activity are provided in the Chelsea Bank program: a post 
decision writing activity that is part of the software and teacher directed activities that are 
outlined in the Teacher Guide. 

After the students make a decision they are asked to write reports on the problem, on the 
banking principle that was the basis for their decision, and on the implications of the decision for 
the bank, the customer, and themselves as teller/customer service representatives. Consistent 
with the goal of reflection, the students may leave the reflection (report writing) at any time and 
return to the decisions or the data in the scenario. The decision itself can be changed at any time 
up until the reflections are completed and submitted. The ability to change the decision at any 
time during reflection is an excellent feature, for it is consistent with the purpose of reflecting. 

We have identified the report writing as decision related reflection and have noted the 
value of being able to re-analyze the scenario or change the decision based on this reflective 
activity. However, that is our inference. Within Chelsea Bank it is simply defined as "report 
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writing" which follows the decision. Classroom Inc. should include some guidance on the 
purpose of the report writing. 

In completing the scenarios, we found the report writing took on a bureaucratic flavor - 
something we had to do simply "because". The questions were not tuned to the issues in the 
scenario and hence they became too vague and unrelated to the thinking actually done in the 
scenario. 

It is not at all clear how this writing activity relates to the objectives. The questions do 
not ask about the problem-solving process or the concepts related to interpersonal skills or the 
banking practice that was the focus of the scenario. In our mind, the questions are too vague to 
support learning of any of the objectives. The writing activity itself supports the learning 
objective of writing. However, we are not so sure that this is a good writing activity. If we are to 
design a writing activity it is essential that the student learn to write to an audience and to focus 
on the purpose for writing. However, in Chelsea Bank there is no purpose nor is there an 
audience for the writing activity. It is simply being done because the program, and the teacher, 
say to do it. 

The second source of reflective activity, the Teachers Guide, provides discussion topics 
more specific to the scenario. These reflective activities are described at the end of each scenario 
under the heading, "After hours: Talking about what happened." The Guide describes the purpose 
of these sections: " These discussion questions make it possible for students to consolidate and 
articulate the decisions and issues they dealt with in the simulation." Many of the questions are 
excellent in doing just that. For example, a question after scenario 4 states: "What skills do you 
think the teller needed in this scenario? Do you think these skills will come in handy in another 
kind of business or in life in general?". We would prefer to see the learners stay in their role at 
this point (e.g., rephrasing the question as, "what skills where most important for you in this 
scenario?") but aside from that, the question addresses precisely the sort of reflection the learner 
should be engaging in. 

Because we think that the After Hours section is a critical component of the learning 
environment, we would encourage the designers to continue to refine this section. Some 
suggestions: 

• Be clear to the teachers that this section is not frivolous, but is central to learning. Explain 
how this activity aids learning. 

• Develop more explicit reflective goals to guide the writing of questions for this section. 
The reflections should center around the concepts and strategies related to the banking 
activity, the interpersonal skills, and the ethics. For the most part the questions are mostly 
focused in interpersonal or customer satisfaction concepts and ethics. 

• Questions should ask students to reflect on what they did, why they did it, what was 
important, how they would improve what they did, and where they can use it in the 
future. 
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• Many questions in this section actually address the consequences of student decisions. 
This reflection on decisions could occur in the context of the report writing and hence be 
a basis for reconsidering the decision. Thus, some of the questions (and puipose) of the 
After Hours section should be considered for integration into the scenario itself. For 
example, the following question, posed in the Scenario 4 After Hours would be an 
excellent question to pose to guide the students in evaluating their decision before it is 
finalized: "What do you think the effect on a company would be if its employees decided 
it was OK to steal from customers they didn't like?" 

Summary and Conclusions: Authentic problems promote some development. 

The goal of this analysis was to assess how the design features of the Chelsea Bank 
software and Teachers Guide support the learning objectives. The learning objectives were 
defined in terms of the areas in which the learner had to make a decision: banking practices, 
interpersonal relations, and ethics. For each of these three areas, there are objectives related to 
understanding the key concepts, developing a problem-solving strategy, and developing the skills 
necessary to implement the strategy. The degree to which the design of Chelsea Bank supported 

these learning objectives was evaluated based on five design features: 

• authenticity of the problem 

• situatedness of the learning 

• scaffolding 

• coaching 

• reflection 

The design analysis indicated that the students are presented with authentic banking 
problems in terms of the banking activities and types of customers. Furthermore, in analyzing 
and making a decision in regard to these problem components, the learners are generally in 
control in investigating and formulating their decision. A positive feature is that there is no 
scaffolding that guides the learner in the decision making — no suggestions as to how the learner 
should proceed in the analysis process. There is little pre-teaching of relevant information or 
hinting as to relevant variables. Thus, with some exceptions, the learner is in control of working 
on the problem and all of the learning arises out of that experience. These two features, the 
authenticity of the problems the learners encounter and the fact that the learners work freely, 
without constraints and generally without pre-teaching, with the banking tools in formulating 
their decision are the two greatest strengths of Chelsea Bank. The students are engaged in an 
authentic activity of solving banking problems. 

A third strong feature of Chelsea Bank is the suspension of the time pressitre normally 
associated with completing a bank transaction, permitting students to collaborate, investigate, 
and evaluate alternative decisions. 

While Chelsea Bank offers a good problem-solving environment, there is not a 
particularly great learning requirement. That is, most of the learning is associated with banking 
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practices primarily focusing on basic skills related to determining if a transaction can be 
completed. More complex materials (bank records, policy manual, etc.) and more complex 
problems would serve to increase the breadth and depth of learning. The problem types we are 
thinking of involve helping customers determine what banking service they need or helping a 
bank teller make a decision on how to treat a customer. In essence, these problems would force 
the learner to think more critically about banking concepts and strategies for achieving customer 
satisfaction. 

The biggest shortcomings in the design of the software is that the learner is provided with 
decision options from which a decision must be made, i.e., there is a multiple choice selection for 
making a decision. This, we suspect, has an impact on the entire learning process in regards to all 
of the objectives. Once the learners know that they must select from a multiple choice decision, 
they will quickly realize that they only need to roughly formulate their decision and then use that 
rough formulation to select among the alternatives. Thus there does not need to be much depth in 
the analysis of the information or in the weighing and debating of alternatives as to what the 
banking problem is and how they should respond to the customer. 

A second shortcoming in the software design is that the problem ends with decision 
making — ^the learner does not follow through with the decision. This has only minor effects on 
objectives related to banking practices but it basically eliminates the learning associated with 
interpersonal skills. Since the learner does not have to interact with the customer there is no skill 
development and, in fact, little demand to develop an understanding of customer satisfaction as a 
concept. This negative consequence for learning interpersonal skills is augmented by the fact that 
the solution to the problems is always to ignore the character of the customer and follow banking 
procedure. 

Perhaps most importantly, there is no guidance for the teachers in how to coach students 
while they work on the problems. Furthermore there is little call for reflection on what has been 
learned. Reflection is primarily in terms of interpersonal skills. We see coaching and reflection as 
essential factors in helping the students to develop strategies for satisfying different types of 
customers and for developing problem-solving heuristics for evaluating banking transactions. We 
also see coaching and reflection as essential in developing an understanding of banking concepts 
and the concepts surrounding customer satisfaction. 

In sum, we find that the design of Chelsea Bank focuses primarily on the learning of 
procedural skills for completing banking transactions. The most important changes in the 
redesign of Chelsea Bank involves eliminating the decision options and providing guidance to 
teachers on coaching strategies overall and in relation to the specific learning objectives. 

Virtually all of the support for learning -- the coaching and the reflection - rests with the teacher 
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rather than the software. We think this is a good feature because it permits discussion of the 
learners' particular understandings and strategies -- the discussion can be situated in the learners’ 
frameworks for thinking about the problem. Thus, it is essential that the teachers are supported in 
their coaching and reflective activities. 
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111. Analysis of the Chelsea Bank Simulations’ Potential 
to Develop the SCANS Competencies 

The U.S. Departments of Labor and Education suggest that teachers provide more 
realistic learning experiences to help students make the school-to-work transition. These 
experiences should model the information-processing, interpersonal, and problem-solving skills 
called for in the new workplace. The Secretary's Commission on Achieving Necessary Skills 
(SCANS) (1991) describes several areas in which students should receive extensive experience. 
Among these are: 

• Resources — ^Identifies, organizes, plans, and allocates resources. 

• Interpersonal — ^Works with others on teams, teaches others, serves clients, exercises 
leadership, negotiates, and works with diversity. 

• Information — ^Acquires, organizes, interprets, evaluates, and communicates information. 

• Systems — ^Understands inter-relationships and can distinguish trends, predict impacts, as 
well as monitor and correct performance. 

• Technology — ^Works with a variety of technologies and can choose appropriate tool for 
task. 

In addition, students should use and develop basic reading, writing, speaking, listening, and 
computational competencies. 

Simulations which involve groups of students in solving realistic workplace problems 
have the potential for developing SCANS-related skills. Chelsea Bank and other Classroom Inc. 
simulations may particularly lend themselves to this, since the simulations are guided by 
computer technology, require students to work in teams, and call for the processing of 
information to solve realistic problems. 

This portion of the Indiana University analysis of the Chelsea Bank simulations addresses 
these questions: 

1 . To what degree do students demonstrate and practice of SCANS competencies while 
using the Chelsea Bank simulations? 

2. What roles do teachers and assignments which parallel the scenarios play in fostering the 
development of SCANS competencies? 

Were the simulations a viable background for developing the competencies? 

The Chelsea Bank simulation places students in the roles of bank branch teller and 
customer service representative. In performing in this role, students appear to be provided 
significant opportunities to use SCANS competencies. For example, students are confronted with 
customers who are pressed for time and are sometimes confused. Some expect certain treatment 
because they are the teller’s friend or expect to be recognized. The students must make decisions 
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about cashing checks, making initial loan recommendations, and providing appropriate bank 
services. This requires asking questions of customers, trying to discern accurate information from 
their life-like answers, which sometimes wander and are occasionally evasive. Success usually 
requires using the computer for looking up credit records, examining the bank procedure manual, 
and perhaps asking further questions. Before decisions can be made, students must discuss and 
evaluate conflicting information with other students in the group. Final decisions about what to 
do are followed by typing into the computer reflections about each decision’s impact upon the 
customer, the bank, and the person making the decision. If a student group successfully resolves 
a sufficient number of teller window problems, its members are promoted to deal with more 
complex customer service problems. 

A common pattern is for teachers to integrate simulations into a semester class plan. One 
or two days a week, students work in groups with the computers. Other days involve work with 
related assignments, such as scripting and shooting a video commercial or writing the dialogue 
and background information for potential new problem-solving scenarios which teach and 
reinforce the curricular goals of an English class, social studies class, mathematics class. In still 
other classes using Chelsea Bank, students have worked on teams to compose directions and tips 
for new employees which they present in the form of manuals and computer-constructed signs. A 
group of students in an economics class participated in a parallel "plan a business" activity which 
involved interviewing neighborhood business people, learning about business plans and loan 
requirements, developing a business plan, and presenting the plan to the class. 

Homework and non-computer days are sometimes focused upon learning concepts related 
to the simulations and parallel assignments such as those described above. After computer 
activity days, classroom discussions sometimes link to other class assignments. For example, in 
one computer simulation, many students mistakenly refuse to cash the check of a poorly dressed, 
smelly individual who actually has an account at the bank. In one teacher's class, this led to a 
next-day discussion of stereotypes and the need to check information. Throughout the semester, 
this classroom discussion served as the basis for several other more traditional activities 
revolving around evidence and support for judgments. 

What methodology was used in the study? 

A team of researchers at Indiana University have been examining student and teacher use 
of the Chelsea Bank simulation in terms of the role the simulation plays in developing SCANS 
competencies. We have interviewed students and teachers, observed classroom activity, gathered 
samples of classroom and homework assignments, and carefully analyzed videotapes of students 
using the computer scenarios. In this section of the analysis, two questions are addressed: 
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1. To what degree do students demonstrate practice of SCANS competencies while using 
the Chelsea Bank simulations? 

2. What roles do teachers and assignments which parallel the scenarios play in fostering the 
development of SCANS competencies? 

Numerous students and teachers were observed on videotape and interviewed as subjects. 

Data for this analysis come from: 

1 . Videotapes produced by MindWorks, Inc. of students using the Chelsea Bank scenarios at 
two schools during the 1994-1995 school year and videotapes of student groups using 
Chelsea Bank scenarios at fo\ir schools during the second semester of 1995-1996; and 

2. Interviews performed by the researchers with students and teachers who have used the 
Chelsea Bank scenarios. 

The students and teachers observed and interviewed came from eight different schools 
using the Chelsea Bank simulation. At some schools there was extensive videotaping of students 
working through the scenarios, and at all schools both students and teachers were interviewed. 
Some of these interviews were performed while on site and others were part of a pattern of on- 
going telephone interviews with instructors. 

A detailed analysis of the presence of activity related to SCANS competencies was 
performed using the videotaped activities of students who had been recorded while using the 
Chelsea Bank scenarios at a high school and a middle school in New York City during 1995. 
From 22 horns of raw video footage produced by MindWorks videos, we selected those 
sequences in which the students were meeting a scenario for the first time. Much of the video 
included staged shots in which students were asked to re-enact their responses to the videos so 
that multiple camera angles could be gathered. None of these shots were included in the SCANS 
analysis. Other imused shots included teacher set-up lectures and attempts to get classes settled. 
Five horns of the original 22 hours was judged to be usefixl for analysis. Nearly all of this five 
horns is accoimted for by 14 sequences showing 13 different student groups. 

Video sequences of students and teachers from fo\ir other schools (recorded during 1996) 
have been analyzed less formally, with an emphasis on the teacher’s role during scenario 
activities. In addition, these teachers have been interviewed about their roles, about student 
progress and reactions to the scenarios, as well as other class activities relating to the simulation. 
At each of these fo\ir schools, several students have also been interviewed about their reactions to 
using the scenarios, what they have learned, and how they have ^plied that learning in and out 
of school. These students range from middle school to upper high school level. 
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The videotaped segments were carefully coded using a variety of instruments. 

Special tools were developed to record the data basic to the study: 

SCANS Codins Forms . The detailed SCANS analysis of the MindWorks tapes was 
conducted using a coding protocol that looked at each minute of scenario activity and rated the 
time spent on every category of SCANS competency and basic skill — ^Resources, Interpersonal 
(student/student), Interpersonal (teacher/student), Information (acquire/organize), Information 
(interpret/evaluate), Information (communicate), Systems (predict/understand), Systems 
(monitor/correct,) Reading, Writing, Math, and Oral. For each minute, the scoring was on a 0-2 
scale, where less than 20 seconds was scored 0, 20-39 seconds was scored 1, and 40-60 seconds 
was scored 2. In addition, any isolated occiurence of a competency (usually allocating Resources 
or noting relationships within Systems) was noted and counted. 

Two researchers coded all of the sequences independently (and achieved 85% inter-rater 
agreement). They then discussed any discrepancies and reconciled them, either from notes taken 
during their separate viewing or by viewing together the relevant minutes again. 

Teacher Interviewer Forms . The interview forms used for teachers asked questions about 
how well the simulation had succeeded in their classes, what they thought students were learning, 
what other activities they were doing in connection with Chelsea Bank, and possible transfer of 
learning to other classes and outside school. In addition to structured interview forms, some 
teachers participated in on-going telephone interviews about their judgments of student progress 
and the sorts of parallel in-class activities they developed in conjunction with the Chelsea Bank 
scenarios. 

Student Interviewer Forms . The interview forms used for students asked questions about 
their enjojnnent of the scenarios, what they were learning, other class activities, and possible 
transfer of learning to other classes and outside school. These interviews were all conducted on- 
site by researchers. 

The data analysis revealed when SCANS categories were being practiced. 

SCANS data analyses mainly produced simple percentage and occurrence reports of the 
degree to which student activity is observed in relation to SCANS categories and in some cases 
subcategories. Definitions of these categories and sub-categories are found in the results 
explanation of this section. Since students performed quite differently when teachers were 
interacting with groups, data are also reported in "teacher present" and "teacher absent" 
breakdown. No one student group was videotaped by MindWorks during early, middle, and late 
scenario use, so it was not possible to legitimately track student growth in SCANS usage as their 
scenario experience increased. Some very cautious hypotheses about student gro\vth were be 
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made, however, comparing SCANS results of student groups videotaped during early scenarios 
with SCANS results of students videotaped during later scenarios. 

Teacher and student interviews were not designed to directly address SCANS use, but 
rather to gather a sense of reaction to the scenarios. It is possible, however, to find several 
examples from teacher and student comments which reveal transfer of learning to SCANS-like 
activities at home, at work, and in other classes. In addition, teacher interviews and observation 
of teacher-developed parallel assignments also occasionally reveal examples of SCANS-related 
activities. Examples gathered from these interviews and observations are informal and serve to 
corroborate and elaborate upon conclusions drawn from the more systematic SCANS analysis 
rather than stand on their own as a separate, structured analysis. 

The results of the study found that SCANS competencies were in use. 

The results of the analysis of the data provided highly informative answers to our two 
major questions: 

To what degree do students demonstrate practice of SCANS competencies while using the 
Chelsea Bank simulations? 

The answer to this question was derived both from the interviews and from the viewing 
of the videotapes. 

Student Interviews : After using the Chelsea Bank scenarios for a semester at John Jay 
High School, 15 students were interviewed about their reactions to the scenarios. Students were 
asked a question which addressed transfer of learning beyond the computer activities to other 
classes and out-of-school situations. Nearly two-thirds of students offered examples of transfer. 
When examples were linked to other school classes, they tended to relate to group work {e.g., 
speaking up more during group-work, using turn-taking and consensus building approaches). 

This relates to the Interpersonal areas of SCANS competencies. Negotiation is one 
subcategory mentioned in this area. Some of the high school students also mentioned directly 
learning some of these skills within simulations. Learning how to work with customers is the 
example most often mentioned: 

• “You learn how to treat people. If you can’t relate to other people, you can’t work 

anywhere. You have to avoid discrimination.” 

• “I learned to separate my feelings from the job rules.” 

• “It helped us imderstand how to control our tempers, calm the customer down, and 

investigate the problem.” 

Some students reported having more sympathy for clerks and bank tellers who need to 
request ID and have to follow specific procedures. Other students reported using information- 
gathering strategies and evidence-based decision making during discussions with their parents. 



ERIC 

hiiiiifHiiffnffTiimJ 




30 



Chelsea Bank Simulation: Analysis of SCANS Competencies 



One young lady reported having fewer arguments with parents as a result. A young man 
described using newly learned group leadership and problem-solving skills when working with 
other youth in the Guardian Angels neighborhood protection organization of which he was a 
member. 

When asked how they had learned to do the transfer activities they described, some 
students mentioned working out procedures on their own while in Chelsea groups. Other 
students mentioned direct instruction, modeling, and reminders from their teacher during non- 
computer time. This sometimes occurred during next-day debriefing sessions. One teacher was 
reported to practice what he preached in showing students how to survey everyone for opinions, 
accept answers without criticizing, and check for consensus. 

Evidence of SCANS competencies on MindWorks videotapes . Videotapes of 13 groups of 
students using Chelsea Bank scenarios have been examined for evidence of students employing 
skills in the various SCANS competency areas {i.e., Resources, Interpersonal, Information, 
Systems, and Technology). The percentages of time are given for 283 minutes of videotaped 
student activity (171 minutes when a teacher is not with the group and 112 minutes when a 
teacher is present). This division is chosen because the students’ activities changed markedly 
when teachers were present. (Because more than one category is usually applicable to any 
activity, the percentages in any column add up to considerably more than 100%.) 



Table 3.1 Student SCANS Activities While Using Chelsea Bank 


Category 


All 


Teacher Absent 


Teacher Present 


Resources 


38 times 


25 times 


13 times 


Interpersonal (student/student) 


42% 


56% 


21% 


Interpersonal (teacher/student) 


15% 


NA 


37% 


Information: (acquire/organize) 


48% 


52% 


42% 


Information: (interpret/evaluate) 


29% 


24% 


37% 


Information: communicate 


18% 


21% 


13% 


Systems: (predict/understand) 


23% 


20% 


26% 


Systems: (monitor/predict) 


26 times 


19 times 


7 times 


Reading 


33% 


39% 


22% 


Writing 


20% 


23% 


14% 


Mathematics 


6% 


8% 


3% 


Oral 


81% 


79% 


84% 
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In order to explain the meaning of each of the categories, they are described below with 
examples of the types of activities that occur in Chelsea Bank videotaped activity. 

• Resources — identifies, organizes, plans and allocates resources: 

...choosing to consult the on-screen manual, memo, or accounts. 

• Interpersonal — works with others on teams, teaches others, serves clients, exercises 
leadership, negotiates, works with diversity: 

...students discussing their decision choices, working together to count cash, and 
cooperating as they write about the effects of a decision (more passive events, such as 
one student reading aloud, are not included); teacher discussing the problem with a 
group. 

• Information — acquires, organizes, interprets, evaluates, and communicates information: 

* acquires/organizes — students reading the start of a scenario, listening to questions 
from the teacher, consulting the manual, and counting cash', 

* interprets/evaluates — selecting from among the decision choices and discussing the 
effects of a decision; 

* communicates — -writing reports on the problem, the effects on bank, teller and 
customer, and the rules for good banking. 

• Systems — understands complex inter-relationships, can distinguish trends, predicts 
impacts, monitors and corrects performance: 

...extended discussions about the impact of decisions and understanding the relationship 
between the bank, the teller/CSR and the customer; 

...brief examples of monitoring and correcting of spelling or pronunciation, or where to 
find information in the manual. 

• Reading 

...the start of a scenario, the materials presented to the teller by a customer, the manual 
and decision options. 

• Writing 

...typing reports on the problem, the effects on bank, teller and customer, and the rules 
for good banking. 

• Math 

...counting cash. 

• Oral 

...reading aloud, discussing decisions, listening to teacher. 

Observations on Videotape SCANS Results . The analysis of the videotapes provides the 
data for some interesting observations. 

• The students (and sometimes teacher) are speaking and listening for 81% of the time, and 
are engaged in interpersonal interactions for 57% of the time. 
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• Processing of information takes place 95% of the time: 48% in acquiring/organizing 
information, 29% in interpreting/evaluating it, and 18% in communicating information. 

• Systems activities (i.e., determining cause and effects) take place 23% of the time. 
Systems incidents take place 26 times, and incidents related to Resources take place 38 
times. 

• Reading occurs for 33% of the time. Writing for 20% of the time, and Math for 6% of the 
time. Both reading and writing occur less often in the presence of the teacher: reading 
decreases from 39% to 22% and writing from 23% to 14%. 

Tentative Hypotheses about Student Growth . Comparisons can be made of students in 
early scenarios (1-4) with other students in middle scenarios (5-8) and later scenarios (9-15). 

• In the absence of a teacher, the oral ratings for the student groups increase from 63% in 
early scenarios to 85% in middle and later ones. This is close to the maximum possible, 
given that a certain amount of time is spent reading or writing with little group 
interaction. Students are learning to discuss at greater length and in more detail. This 
increase does not occur when the teacher is present and activity tends to be more teacher- 
led. Further discussion of this follows below. 

• Ratings of student reading increase from 27% in early scenarios to 45% in middle and 
later ones. Students are using reading to search for information and to help evaluate 
earlier information. This increase does not occur when the teacher is present and students 
tend to look to the teacher for answers. 

Conclusions about student growth must be very tentative since it was not possible to track 
a single set of students all the way through the scenarios. The general trend of increased SCANS 
activity is, however, encouraging and merits more careful examination with systematically 
videotaped student groups. 

It is clear that students using the scenarios in the MindWorks videos employ and practice 
SCANS problem-solving competencies a good deal. This is confirmed by student interviews 
which identify applications beyond the computer activities and acknowledge the roles played by 
both the scenarios and teacher direction in producing learning. Chelsea Bank activities viewed in 
the videos require more active student involvement with SCANS competencies than do more 
traditional listen-to-lecture, read-the-chapter, and answer-end-of-chapter question activities. 

What roles do teachers and assignments which parallel the scenarios play in fostering the 
development of SCANS competencies? 

The role that teachers play in shaping the Chelsea Bank experience while students are 
using the computer programs appears to be very important. There does not, however, appear to 
be a clear pattern for how teachers interact with students. The tentative conclusions below are 
drawn from careful analysis of the extensive videotaped interventions of two teachers on the 
MindWorks tapes and, to a lesser extent, from videotapes of four teachers who used the Chelsea 
Bank scenarios during Spring of 1 996. 
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Data from the SCANS analysis of the MindWorks tapes indicates that students are more 
likely to interpret and evaluate information when teachers are present (37%) then when alone 
(24%). In actual fact, this higher interpreting/evaluating result is due mainly to the persistent 
urging of one videotaped teacher for students to ask evaluative questions. The second teacher 
intervened to ask more factual level questions and students actually did more evaluation when 
she was not present than when she was. This difference between teachers and the subsequent 
difference in student performances are very important. 

The four additional teachers who appear on more recent videotapes seem to differ as 
much from the two MindWorks teachers as these two teachers differ from each other in terms of 
their relationships to students doing the scenarios. Three of the additional four teachers appear to 
only intervene in student discussions when summoned by students. Interactions and feedback 
tend to be directive. For example, teachers tell students step-by-step how to solve a computer 
problem, or in some cases they direct students on how to solve scenario problems. Neither 
evaluative nor factual level questions appear to be much in evidence. Only one of these four 
teachers is videotaped initiating interaction between himself and his student teams while they 
worked on computer scenarios. This teacher frequently approached a group, listened to the group 
members’ discussions and observed their progress for a few moments, and then offered 
unsolicited guidance and advice to the team. 

Student interviews . As described earlier, several students were interviewed about their 
evaluations of the Chelsea Bank experience. When describing memorable recollections, several 
mentioned parallel assignments, such as scripting and producing videotaped commercials based 
upon bank services or scenarios. When probed to explain how they had learned to transfer what 
they had learned beyond the class using Chelsea Bank, some students gave credit to direct 
guidance from their teacher during regular government class meetings. The teacher was reported 
to describe and model ways to survey all group members, avoid criticism, and give all group 
members credit for what they did. Out-of-school transfer examples were more diverse. 

Teacher interviews and observations . Interviews with teachers revealed a wealth of 
activities which had been developed to parallel the Chelsea Bank scenarios. Among these were: 
1) scripting and shooting a video commercial; 2) writing dialogue and backgroimd information 
for potential new problem-solving scenarios; 3) composing directions and tips for new 
employees in the form of manuals and computer constructed signs; 4) planning a business by 
interviewing neighborhood business people, learning about business plans and loan 
requirements, and developing and presenting a business plan; 5) working in class on banking 
activities like: accepting checks, writing checks, using deposit and withdrawal forms, bank 
balance recording sheets, and reconciling statements; 6) writing advertisements and job ads for 
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teller supervisor and other positions; 7) visiting a local regional Reserve Bank where students 
saw money being shredded, xmderstood explanations of Reserve Bank activities, and asked 
relevant questions. 

Teacher interviews also revealed strikingly different conceptions about what students 
were to be learning and how they were to go about it. Some teachers described the scenarios as 
useful activities in their own right which would somehow teach students about the real world. 

The role of the teacher was seen as keeping students on task and giving them needed boosts to 
help them get through the scenarios in the class time available. Help was provided when solicited 
or when students appeared to be in difficulty. Other teachers saw the scenarios as another venue 
to practice the social and group dynamics lessons being taught as part of classroom goals. 
Modeling, direct instruction, and reflection were natural parts of all activities. 

Still other teachers saw their role as being deliberately hands-off so students didn't 
become too dependent. These teachers provided very little interaction in most scenarios, 
confining themselves to answering student questions in order to set them moving again. Still 
other teachers saw themselves as providing the necessary mental stepping stones for students to 
think correctly about how to solve the problems. One teacher, for example, was observed 
describing the scenario in detail before students started (even saying, “And maybe he’s made an 
error in counting the money.”) His comments during scenario tended to keep students moving 
through the activity more than thinking about issues. 

What can be concluded then about the program as a vehicle for developing 
SCANS competencies? 

Videotape data from this study demonstrate wide-spread student use of SCANS 
competencies when learning with Chelsea Bank scenarios. Interview data from students and 
teachers corroborate transfer of these learned competencies from the scenarios to use in other 
classes, at home, and at work. Though the videotape coding system is capable of tracking some 
indices of quality {i.e., various levels of processing information and use of Systems), for the most 
part the SCANS video analysis data is not a good indicator of the quality of student competency 
use. For example, the system is not currently capable of discerning improvements in reading 
abilities or changes in the quality of interpersonal interactions. It only tracks, in accurate detail, 
the fact that reading and interpersonal activity are occurring. In some cases, for example, reading 
involves students struggling to make sense of a single word and in other cases students are using 
several sophisticated strategies to read from multiple sources. Forty seconds of either activity are 
counted equally. 

Interview data provide examples of high quality transfer of SCANS learning for some 
students and help add detail to the picture of what occurs when students learn with Chelsea Bank. 
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This interview data, however, is not systematically gathered from a cross section of students and 
teachers and cannot allow us to draw solid conclusions about general impact. Teacher 
observation/rating scales (based on video data gathered during the first year of this study) are 
being developed to better track the quality and improvement of student SCANS competency use. 

Teachers have been able to develop a wide range of assignments which parallel Chelsea 
Bank scenarios and appear very likely to extend and expand student use of SCANS 
competencies. These assignments are imaginative, challenging, often draw upon real-world 
resources, and build bridges to the classes with which Chelsea Bank is integrated. Other teachers 
using Chelsea Bank could benefit from a systematically gathered and organized collection of the 
best of these activities. 

Although all teachers received essentially the same preparation for using the scenarios, 
the video data, interview data, and classroom observations all indicate significant teacher 
differences in impact upon students. Further, teacher understanding of what their roles ought to 
be also differ considerably — and these differences appear to be important. When some teachers 
intervene with students, the quality of student information processing consistently rises while 
with other teachers information processing consistently declines to a more factual level of 
following directions. Some teachers perceive their roles to include modeling and guiding 
students. They use probe questions, explicit instruction in how to function in groups, and model 
during regular classes the methods by which students should respectftilly survey each other for 
information and reach consensus to solve problems. Other teachers see themselves as needing to 
take a more "hands-off' role except in removing obstacles which inhibit students from using the 
scenarios. Still other teachers seem to see their role to be that of "hint giver" or "encourager" or 
"technician." 

Each of these roles may be important and necessary, but it seems clear that student 
growth in particular SCANS areas {i.e., information processing, interpersonal functioning, 
understanding systems, etc.) is strongly related to appropriate and effective teacher use of these 
roles. For this reason, it is very important to study further ways that teachers perceive their roles 
in relation to Chelsea Bank and the development of the SCANS competencies. Since teacher 
interventions (during scenario use and in developing parallel activities) appear to make so much 
difference, it is also important to examine and develop methods for expanding and supporting 
effective teacher activity when using Chelsea Bank and other Classroom Inc. programs. 
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IV. Analysis of Problem-Solving Behaviors as a Guide 
To Developing Assessments 

The year-one study focused first on the SCANS competencies as outlined in the 
U. S. Department of Labor Secretary ’s Commission on Achieving Necessary Skills 
(SCANS) (1991). This analysis (see Section III) was important in order to understand 
whether the lessons were developing the competencies described in that report; however, 
the researchers felt that those competencies were not detailed enough. It was necessary to 
define them more clearly as problem-solving behaviors that could be evaluated with the 
assessments that were to be developed. 

What structured our analyses of the program? 

Readers’ cognitive and metacognitive strategies had been identified by the 
researchers in previous studies. These studies resulted in a list of “thinking” or “problem- 
solving” strategies that readers engaged in as they constructed meaning. In many ways 
the list of reading strategies resembled the typical set of problem-solving strategies that 
have been described in the research and theoretical literature; and it was, therefore, 
modified to more directly reflect that backgroimd. This modified list was then used in 
initial analyses of students using the Chelsea Bank simulations, and it evolved further as 
data was collected fi'om viewing the videotapes. 

Thus, as we analyzed the behaviors of those using the program, their performance 
was helping to structure the refinement of the categories for the assessment to be 
developed in years two and three. The students’ problem-solving strategies and the 
teacher behaviors that seemed to be influencing student performance became an 
important focus of this phase of the study — as were several key strategies that experience 
and the literature endorsed even though they were not prevalent on the tapes. The 
procedures used in the study are discussed in this section of the report with a description 
of how the videotapes were segmented for analysis. The categories are then defined, the 
reliability of the ratings using them is presented, and the findings based on them are 
reported. 

The intensive analysis of problem solving had a very practical objective: It 
underlined and helped articulate the need for an assessment component in the program, 
and the understanding and description of student behavior which grew out of the study 
will be used as a basis for developing student assessments. At the same time, it should 
guide the teacher’s use of the program — a need underlined by a key observation about the 
influence of teacher presence. This observation is discussed following the findings based 
on major categories and problem-solving strategies. Also, other aspects of student 
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behavior were analyzed and are discussed in a comparison of grade levels and a 
consideration of the impact of particular scenarios and their sequencing. A more detailed 
discussion of the students’ use of the categories and strategies examined follows with a 
description of four segments on the videotapes. 

As the report will demonstrate, the study rationalized the aims proposed for years 
2 and 3 of the study. The need for an effective assessment component for the Chelsea 
program is discussed as a background for the concluding sections of this report, 
which detail this study’s impact on the approach and design that will be used to build the 
assessment component. 

The study has clearly defined objectives for the assessment. Included in these is 
the goal of developing assessments that will focus learning, that will meet goals dictated 
by several major categories that have guided our analyses, that will develop the problem- 
solving strategies the study has verified as important student behavior, and that will help 
students and teachers make more effective use of the Chelsea Bank simulations. 
Hopefully, too, in helping the use of the program to flourish, the assessment component 
can contribute to it as an impetus to school reform. 

How was this study grounded? 

This phase of the study began with a look at four general categories that had been 
emphasized and described by program developers at Classroom, Inc. These seemed 
implicit foci for the program and provided a first cut for the analysis before the 
researchers moved to a more specific analysis of students’ problem-solving strategies. 

The description of these four general categories was somewhat redundant of the 
SCANS analysis that is reported elsewhere in this report. This redundancy was plaimed 
and helped to provide a validity check for the two analyses. However, this phase of the 
study used these four general categories only as a starting point for further analysis. In 
addition, this phase did not look at all of the SCANS competencies as was done in the 
study reported in Section III. 

These four general categories included the focus on basic communication and 
mathematics as a major category labeled Use of Basic Skills. A second major emphasis 
was on Problem Solving as a focus on attempting to solve — ^to reach a solution to — a 
posed problem. A third general category was Collaboration, which might better have 
been called “group work.” It did not attempt to analyze whether students actually 
collaborated in arriving at a solution or understanding a problem. Rather, at this general 
stage, the question was whether students were working together cooperatively to 
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complete a task. The fourth general category examined was whether the students were 
developing competence in the Use of Banking Concepts. 

The literature links problem solving and critical thinking. 

The reading strategies identified in previous studies by Farr and Greene were used 
beyond the starting point just described in developing the initial list of strategies to be examined 
in the videotapes. This list of strategies had been analyzed in relation to similar studies reported 
in the literature. The literature on problem solving and critical thinking is extensive and cannot 
be reviewed in this paper. However, a consideration of a few of the major examinations of 
problem-solving theory and of critical thinking analyses will demonstrate some of the influences 
on the final list of problem-solving strategies that were used in this study. 

Critical thinking is often linked directly to decision making, so that the two become 
interwoven and frequently synonymous. The influential work of Norris and Ennis (1989) makes 
this connection with five steps involved in critical thinking: 

1. Clarify the issue by asking critical questions, 

2. Gather critical information about the issue, 

3. Begin to reason through the various points of view, 

4. Gather further clarifying information and conduct further analysis as needed, and 

5. Make and communicate the decision. 

The eight strategies that were identified by this study for the viewing and analysis of the 
videotapes rely heavily on this sequential description of what is, in effect, issue resolution or 
problem solving. In order to conduct this kind of rational behavior, the student using the Chelsea 
program must focus on the problem (step 1) and rely on previous specific and general 
learning (step 2). Steps 2,3, and 4 of the Norris-Ennis model also relate to recognizing when 
one is confused, seeking help for clarification, and making changes in one’s thinking — one’s 
assumptions, predictions, and options for resolution. These activities, which were strategies 
identified in this study, are also relevant to steps 3 and 4 above, as is another key strategy 
identified by our study: collaboration in solving the problem. That strategy is also related to 
Norris and Ennis’ last step, as is our general category of collaboration. 

This kind of overlapping and interlacing is not uncommon when one begins comparing 
different theoretical descriptions of critical thinking and problem solving, and it is equally 
prevalent across models based on the same seminal thinking, such as that of Norris and Ennis. 
The descriptions and categorical schemes that are developed in different studies to fit particular 
learning environments echo the operational, process-relevant emphases in Norris and Ennis. 
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Like the many applications of the SCANS competencies, categorical schemes and 
analysis designs take the general descriptions and modify them in applying them to specific 
situations — as we have done in studying the Chelsea Bank program. 

Another example of the kind of work that provided background for our analysis is that of 
Marzano (1992). In particular applications, this model has been described with many more 
facets, but it is built on five key dimensions'. 

1. Positive attitudes and perceptions about learning, 

2. Acquiring and integrating knowledge, 

3 . Extending and refining knowledge, 

4. Using knowledge meaningfully, and 

5. Productive habits of mind. 

The similarity of Marzano’s second and third dimensions to steps 1, 2, 3, and 4 of the 
Norris-Ennis general model is obvious. They are also compatible with the processing strategies 
used in our study. Clearly the need to focus on a problem while using previous learning, seeking 
help, recognizing confusion or the need for clarification, and making changes cross all of 
Marzano’s dimensions, particularly numbers 2 and 3. Dimension 4 is a general classification that 
includes the prediction-making, inference-drawing strategy that we have valued highly as a 
problem-solving behavior. As Marzano details them, dimensions 1 and 5 cover our concern with 
being focused, the value that the Chelsea Bank program places on collaboration, and the 
strategies of recognizing confusion, getting help, and making changes. While these behaviors — 
like revision of one’s thinking and expression in writing — are not often prevalent in the student 
behavior viewed, they are, we have concluded, endorsed by many of the foundational 
descriptions of problem solving. 

The highly useful Marzano model exemplifies another characteristic frequently foimd in 
theoretical models of thinking processes: Its first and last dimensions are, from a logical 
perspective, not viewed from the same perspective as the other three. Although dimension 5 is 
detailed with some specific recommendations, it appears more attitude-relevant than strategic, 
like steps 2-4. What brings the dimensions together is that they are all broadly scoped goals for 
students w'ho are thinking critically in order to solve problems; thus they are, like the Norris- 
Ennis model and the 12 categories used in this study, a reasonably operational attempt to 
describe the behavior. 

Typical of the process-oriented descriptions that ensue from such models is one by 
Fogarty and Opeka (1988), which includes active listening, articulation, cooperation, 
questioning, organizing and analyzing data, comparing, contrasting, classifying, and creative 
strategies such as brainstorming, visualizing, personifying, inferring, and using analogies. 
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Numerous strategies like Fogarty and Opeka’s and one relating problem solving directly 
to mathematics (Sackett, 1994) are more common than short lists. Sackett says that sensitivity, 
cooperation, and collaboration are needed to solve mathematical problems. The competencies he 
lists are communications skills — treading, writing, and speaking; learning skills, including use of 
information; and critical thinking skills like the analysis of facts. Yet in a summary of critical 
thinking skills, Potts (1994) presents only three general strategies, one of them the traditional 
inductive behavior of forming categories, one the ability to find or identify problems, and one 
“enhancing the environment.” The latter seems more directed to the teacher, but it does stress the 
attitudinal factors present in other models. 

The methodology begins with a belief in metacognition. 

The methodology used in this study assumes that the students using the Chelsea Bank 
program have some metacognitive sense how they are processing the information presented in 
the scenarios and how they are coming to a decision that is a solution to the problem presented in 
each simulation. Developing metacognition in students has become a major emphasis in the 
theory-making and research on learning and thinking. A keystone in the metacognitive literature 
is the work of Flavell (1979), who found only limited ability in very young children and more as 
the age of his subjects increased. This suggests that metacognitive processes are learned. 

Many researchers have pursued the delineation of metacognitive strategies among 
children and have considered ways of developing them. Bareli (1995) includes a chapter on 
developing metacognition in an extensive book on strategies for teaching thinking. Like Bareli, 
Fogarty (1994) believes that metacognition involves not just awareness of one’s own thinking 
behaviors and strategies, but control of them as well. With James Bellanca, Fogarty (1991) links 
the development of metacognition directly to cooperative team learning. Krulik and Rudnick 
(1994) promote metacognition as a means of self evaluation. 

The observational methodology profits from verbal protocol analysis. 

Depending on the metacognitive process, and assuming that it will operate within the 
Chelsea Bank program, the methodology of our study is grounded also in verbal protocol 
analysis. This research technique is most often conducted as some situation which promotes 
thinking aloud and hopefully data which can be analyzed to reveal how the student is reading, 
thinking, or, in the case of this study, problem solving. The theoretical base for it is found in the 
work of Ericsson and Simon (1993). Ironically, perhaps, they summarized much of the work 
done and the relationship of protocol analyses to information-processing theory — with a 
somewhat cautious, if not pessimistic, outlook on its potential. Pressley and Afflerbach (1995) 
have also published a relatively complete review of the methodology acknowledging its strengths 
and the challenges it presents the researcher. They offer guidelines and explicit instructions on 
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how to make it most effective. The researchers of this study are thoroughly familiar with the 
methodology and most of the studies reviewed in these two valuable reviews. They have 
conducted several studies using and substantively developing the technique {e.g., Farr, Greene, 
and Pritchard, 1992). 

The method is somewhat different from that used in the analysis of our study because the 
students were not cued directly by the researchers to reflect on the processes they were using to 
solve the problem. The Chelsea Bank program, on the other hand, does cue self evaluation of 
decisions made by directing the students to articulate the consequences of the action they take — 
from several perspectives, including the customer’s, the students’ as bank employees, and the 
bank’s. 

In addition, it is important to note that we found that the nature of the Chelsea Bank 
program tended to produce data not unlike that generated by verbal protocols. The team- 
structured approach to making a decision and to generating interactive comments as responses to 
the program led to the students sometimes explaining their reasoning to each other, creating the 
kind of data that is generated by introspective verbal protocol. It also produced verbalized 
reactions to the input of teammates. One might have expected that when the students turned to 
their teachers for help, the exchange would have approximated the kind of cueing and student 
self-analysis produced in verbal protocol recall. This, however, was not often the case, as the 
teachers often either merely gave directions or referred the students to the tools of the program, 
which, as already noted, did tend to serve as a kind of prompt to self-analysis. 

Thus, while the articulation of student thought and action provoked by these aspects of 
the program and recorded on the videotapes did not equal the type of data produced by think- 
alongs per se, it came close in the eyes of our viewers, who are highly trained and experienced in 
a methodology they have helped develop and refine. We felt that our methodology, which 
combined observation and verbal protocol analysis, was effective and tended to preclude the kind 
of problems that develop in introspective verbal protocol from the interruptive nature of the cues 
that produce the self-reporting. Nor was it subject to the reliance on memory to recall thoughts or 
reactions after the student performance, as is the case with retrospective verbal protocol . 

In addition, the methodology used in this study did not rely on cues that might, to the 
student, seem to require thoughts or responses that the student might not otherwise have. The fact 
that the students were being videotaped may have affected their performance — ^perhaps to inhibit 
them and limit their verbal input — but the verbalizations studied were responses to the program, 
not to the researchers. Thus approaches useful in verbal protocol analysis could be adopted to 
look at the metacognitive data produced by the students. This enabled us to rate the four major 
categories and eight problem-solving strategies. 
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How was the methodology applied? 

The primary methodology utilized for this phase of the Chelsea Bank study entailed the 
analysis of 22 hours of videotaping of junior high students from I.S. 218 and eleventh graders 
from John Jay High School. The videotaping was produced during the 1994-1995 school year. 
Classroom Inc. provided a log manual, containing a detailed description of the duration of the 
tapes in addition to the names of all student groups, and interviews with both teachers and 
students. 

The first phase of the study consisted of an analysis of students on the videotapes. Before 
this could take place, the 22 hours of videotaping were pre-screened and reduced to 6 hours and 
23 minutes. At this point, the teacher and student interviews were eliminated because the 
research team was solely interested in the students' problem-solving interactions. Further 
refinement of the time intervals was performed. During this revision, inaudible time intervals and 
retakes (reversals) were discarded. Thus, there were approximately 6 hours for the final analysis. 

The time intervals of the students' interaction were found in 25 different videotapes. In 
order to facilitate the viewing of these tapes, the team decided to compile the time intervals into 
one single VHS tape. 

The videotape was viewed in episodic segments. 

The analysis was conducted on the basis of segments. A segment was defined as a unit of 
activity in which the students were focused on a particular activity. The segment began with the 
initiation of the video and/or an activity the students were attempting to accomplish. When the 
students completed the activity and went on to a different activity, the segment was concluded. 
Sometimes the teacher would interrupt the students and focus them on another issue. This would 
also indicate a segment change. It would be too general to merely describe a segment as a unit of 
activity, but that is exactly what it was. Each segment had a clear beginning. The segment 
concluded as the dynamics of the interaction shifted. For example, students might be working on 
a scenario, focusing on a customer's account. The segment would end if the teacher interrupts 
their discussion by having them focus on a different feature. 

We chose to analyze the four general categories and the eight problem-solving strategies 
by segments since we were interested in the behaviors the students engaged in as they worked 
through the scenarios. We did not want to segment the tapes into arbitrary time units since 
collaboration and problem-solving interactions would have been lost. This segment analysis 
based on episodes of behavior has been used by Corsaro (1981) and others who have analyzed 
children’s play activities. Corsaro argues that the use of episodes rather than time slices for 
analysis allows the researcher the opportunity to analyze events which often depend on preceding 
and subsequent behaviors in order to be understood. 
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Because of the selection of episodes rather than time segments, some of the segments are 
quite long and others are quite short. The unevenness in the length of the segments is a limitation 
in the analysis and is a trade-off for getting more contextually valid analyses. 

A total of 50 segments were identified across the 6 hours of video which had been 
prepared for analysis. The segments vary in length as some are very short, lasting a minute; 
others are as long as 25 minutes We conducted independent analyses of the division of the 
videotapes into segments. The researchers agreed in almost every case as to the be ginnin g and 
end of a segment. Discussion clarified the minor differences in segment division that were found. 
Table 4.1 provides descriptive information for these 50 segments. Table 4.2 summarizes the 
different scenarios that were included within these 50 segments. This summary was used as a 
basis for choosing two scenarios for comparison and is reported in a later section. 



Table 4.1 Video Segments Selected and Analyzed (N=5Q) 



School 


Tape Number 


Start Time 


End Time 


Total Time 


Scenario 


IS 218 


36 


38.09 


49.47 


11.38 


11) Ms. Austin 


IS 218 


36 


37.13 


38.08 


00.55 


11) Ms. Austin 


IS 218 


36 


30.57 


37.12 


06.15 


11) Ms. Austin 


IS 218 


25 


30.44 


39.19 


08.35 


10) Mr. Smith 


IS 218 


25 


39.20 


43.49 


04.29 


10) Mr. Smith 


IS 218 


24 


47.33 


58.00 


10.27 


10) Mr. Smith 


IS 218 


23 


00.50 


17.22 


16.32 


9) Mr. Jones 


IS218 


16 


22.30 


31.05 


08.35 


8) Mr. Smith 


IS 218 


16 


09.03 


22.28 


13.25 


8) Mr. Smith 


IS 218 


13 


00.47 


11.49 


11.02 


7) Mickey Mike 


IS 218 


13 


22.05 


31.30 


09.25 


6) Dooley 


IS 218 


12 


09.07 


12.32 


03.25 


6) Dooley 


IS 218 


12 


12.33 


30.03 


17.30 


6) Dooley 


IS 218 


6 


01.03 


06.32 


05.29 


2) John Gumble 


IS 218 


5 


08.21 


31.13 


23.52 


2) John Gumble 


IS 218 


3 


03.19 


04.32 


01.13 


1) Fran Friendly 


IS 218 


3 


24.29 


25.36 


01.07 


1) Fran Friendly 


IS 218 


3 


20.29 


22.12 


01.43 


1) Fran Friendly 


IS 218 


3 


17.25 


20.02 


02.37 


1) Fran Friendly 


IS 218 


3 


04.45 


06.29 


01.44 


1) Fran Friendly 


John Jay 


38 


30.50 


36.08 


01.18 


15) Ms. Highland 


John Jay 


37 


46.49 


60.00 


13.11 


15) Ms. Highland 


John Jay 


32 


30.50 


42.00 


11.10 


11) Ms. Austin 


John Jay 


31 


30.47 


35.29 


04.43 


10) Ms. Smith 


John Jay 


31 


51.26 


59.52 


08.26 


11) Ms. Austin 


John Jay 


30 


31.43 


36.40 


04.53 


1 0) Mr. Smith 


John Jay 


30 


36.41 


43.40 


06.51 


1 0) Mr. Smith 


John Jay 


30 


43.41 


59.22 


15.41 


1 0) Mr. Smith 


John Jay 


30 


30.30 


31.42 


01.12 


10) Mr. Smith 


John Jay 


29 


36.40 


40.40 


04.00 


1 0) Mr. Smith 


John Jay 


29 


30.50 


36.38 


05.48 


9) Mr. Jones 


John Jay 


29 


56.27 


58.59 


02.32 


10) Mr. Smith 


John Jay 


28 


49.02 


59.23 


10.21 


9) Mr. Jones 
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Table 4.1 Video Segments Selected and Analyzed (N=50) 


School 


Tape Number 


Start Time 


End Time 


Total Time 


Scenario 


John Jay 


20 


00.47 


04.31 


03.44 


8) Mr. Smith 


John Jay 


19 


13.48 


26.27 


12.48 


7) Mickey Mike 


John Jay 


19 


26.28 


29.40 


03.12 


8) Mr. Smith 


John Jay 


18 


00.45 


10.12 


09.27 


8) Mr. Smith 


John Jay 


18 


10.13 


26.35 


16.22 


6) Dooley 


John Jay 


11 


00.43 


04.16 


03.33 


5) John Beadle 


John Jay 


10 


26.10 


30.43 


04.33 


4) Jane Smiley 


John Jay 


10 


07.22 


26.00 


16.38 


4) Jane Smiley 


John Jay 


8 


00.08 


05.34 


05.26 


2) John Gumble 


John Jay 


7 


06.37 


12.45 


06.08 


2) John Gumble 


John Jay 


7 


13.16 


17.25 


04.09 


2) John Gumble 


John Jay 


7 


17.26 


31.05 


12.41 


2) John Gumble 


John Jay 


1 


11.54 


14.07 


02.13 


1) Fran Friendly 


John Jay 


1 


25.25 


27.41 


02.16 


1) Fran Friendly 


John Jay 


1 


23.37 


25.24 


01.37 


1) Fran Friendly 


John Jay 


1 


21.12 


23.28 


02.14 


1) Fran Friendly 


John Jay 


1 


16.14 


19.12 


02.58 


1) Fran Friendly 


Total observation time 6 hours, 3 minutes, 23 seconds 



Table 4.2 Summary of Video Segments Selected and Analyzed 

(N=50) 


Scenario 


Number of Segments Viewed 


Total Time Viewed 


1) Fran Friendly 


10 


91 minutes, 53 seconds 


2) John Gumble 


6 


39 minutes, 34 seconds 


4) Jane Smiley 


2 


16 minutes, 32 seconds 


5) John Beadle 


1 


3 minutes, 12 seconds 


6) Dooley 


4 


33 minutes, 55 seconds 


7) Mickey Mike 


2 


22 minutes, 4 seconds 


8) Mr. Smith 


5 


27 minutes, 27 seconds 


9) Mr. Jones 


3 


6 minutes, 43 seconds 


10) Mr. Smith 


10 


68 minutes, 5 seconds 


11) Ms. Austin 


5 


38 minutes, 48 seconds 


15) Ms. Highland 


2 


1 1 minutes, 44 seconds 



The categories used for the analysis were fine-tuned. 

As the analysis of the tapes to identify student and teacher behavior began, three raters 
worked together on the first five segments to reach to a consensus for rating purposes. These first 
five segments were reviewed and re-analyzed many times. This analysis, re-analysis, and 
discussion resulted in a number of changes and clarifications in the list categories and strategies. 
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I 
I 

For example, the “concepts” category was first called “content.” We found that this term was too 
broad and after much discussion arrived at the term concepts, which was defined as the 
discussion and application of terms having to do with banking, such as withdrawals, loans, 
deposits, and signature cards. A problem-solving behavior that seemed relevant when the study 
began was “visualization,” whether the problem solvers describe a scene or event. This behavior 
has been discussed in the reading literature as readers constructing mental images. The literature 
on problem solving also mentions this behavior, although it has not been a major focus. Because 
of the use of the computer as a visual stimulus, references to visual images were not found in the 
students’ verbalizations. After much discussion, this category was dropped. 

The literature on problem solving and meaning construction is replete with references to 
the use of previous knowledge. We had also seen this expressed in our analysis of reader 
behaviors in previous studies. As soon as we began examining the tapes, we realized we were 
able to divide this category into the use of previous specific knowledge and the use of previous 
general knowledge. Specific knowledge was defined as knowledge and skills that had been 
specifically learned in the Chelsea Bank simulations, while general knowledge was knowledge 
and skills that were learned prior to (or outside of) the Chelsea Bank simulation and were 
brought to bear on solving the Chelsea Bank simulation problems. 

Two problem-solving strategies that received a great deal of attention in our initial 
analyses and discussions were making predictions and making changes. We felt that the literature 
on problem solving and critical thinking strongly endorsed the notion that problem solving meant 
putting forth trial answers and modifying them as a sequenced response. We had also observed 
these behaviors in our studies of readers as they constructed meaning from text. However, we 
saw so little of either of these behaviors in our initial analyses that there was some thought given 
to the possibility of dropping them from the analysis. We did not do so because we concluded 
that both of these behaviors are crucial aspects of problem solving and critical thinking. 

In order to clarify the difference between collaboration as one of our four general 
categories and collaboration as one of eight problem-solving strategies, we debated the 
advisability of changing the name of one of the categories. We did not do so, but we spent much 
time clarifying the differences between the two until each member of the research team felt the 
distinctions were quite clear and understandable. Collaboration as a general category was the 
concern as to whether students were able to work together cooperatively. This did not mean that 
they were collaborating in arriving at an analysis of the problem or whether they collaborated in 
developing a response. Rather, we wanted to have a general look at whether they worked 
cooperatively. On the other hand, when we got to the level of the problem-solving strategies, we 
were interested in whether the collaboration went beyond merely working together. We wanted 
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to know if they developed shared, jointly developed, responses to questions. We wanted to know 
if they truly cooperated in solving the problem. They could have been quite high in collaboration 
as a general activity (working together), but not as high in putting their heads together to solve a 
problem. 

Another duplication in labels that necessitated clarification was that of problem solving as 
a general category and problem solving as the overall label for the eight problem-solving 
strategies. As we conducted our general analysis, we wanted to be sure to capture evidence that 
the students were engaged and focused on seeking the solution to a problem. Anyone who has 
examined the tapes and observed students engaged in the Chelsea Bank simulations comes away 
with the feeling that the students are engaged in the pursuit of an answer to a problem. We 
wanted to be sure that our fct general analysis captured this data as it existed in the tapes. On 
the other hand, we wanted to be able to analyze at a deeper level just what the students were 
doing as they engaged in the pursuit of answers to problems, i.e., what kinds of problem-solving 
strategies did they seem to be using. 

After these many reviews and discussion, the three reviewers felt they were in general 
agreement as to what they were looking for in the videotapes and how the behaviors would be 
classified. The clarification of definitions, the discussion of specific examples of the behaviors, 
and the deletion and addition of several behaviors, resulted in a usable framework for moving 
ahead with the analysis of the remaining forty-five segments. A list and definition of the four 
general categories and the eight problem-solving strategies is provided below. 

The viewers’ analysis scheme included four general categories. 

The four general categories of analysis established four descriptors for analyzing each 
segment: 

Basic Skills . This descriptor indicates the extent to which students are engaged in an 
activity which causes them to use basic skills — including mathematics, reading, and writing. 
(Listening and speaking are not included since these are constants of most of the activities.) This 
category includes such things as students reading from the screen, the resource manual, and any 
other materials. It also includes students writing notes, questions, and ideas about the simulation 
activity or typing responses on the keyboard. 

Mathematics includes any manipulation of numbers. This could be the actual writing 
down of numbers, developing tally sheets, or doing any other mathematical manipulation 
whether written down or expressed verbally. It does not include listening and speaking. As much 
as we believe these are basic skills, there was so much of this in evidence that these would have 
dominated the analysis. 
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In addition, computer skills such as keyboarding are not included. The SCANS analysis 
did focus on these behaviors, and we were interested in this part of the analysis in the more 
traditional basic skills as assessed on most state and school district assessments. 

Collaboration . This descriptor indicates the extent to which students are working together 
during the segment and exchanging information and points of view in order to develop joint 
responses. All of the behaviors that would be associated with cooperative learning are included 
under this heading. Cooperation in sharing tasks such as the use of the mouse, entering data, or 
keeping notes are included. (Often times the students were working individually but they were 
cooperating as they did so. That is, the students allowed each other to read and would ask if each 
other had finished the reading of the screen. They worked cooperatively, but there may have been 
little collaboration in solving problems and discussing issues.) 

Problem Solvins . This descriptor indicates the extent to which students are attempting to 
solve a problem. This category includes evidence as to whether the students were focused on 
working towards the solution to a problem. This does not mean that they had identified the 
problem, but merely that they recognized that they were to solve a problem and that they needed 
to “dig in” and do some things to solve the problem. Another way to consider this category was 
whether the students were more than just focused. It is possible to be focused and engaged, but 
not be working towards the solution to a problem. This category attempts to capture the extent to 
which students were engaged in, and focused on, solving a problem. 

Concepts . This descriptor indicates the extent to which basic concepts regarding banking 
or the general world of work are being emphasized. The Chelsea Bank simulations exposed 
students to a number of banking concepts and basic banking information. We wanted to 
determine the extent to which the students were using and seemingly understanding such things 
as the processing of loans, approving and cashing of checks, and other such concepts. We felt it 
was important to know if a simulation activity such as Chelsea Bank fostered student use and 
apparent understanding of basic concepts that were new to them. 

Problem-solving was analyzed using eight strategies. 

The final scheme for analyzing the problem-solving behavior of the students videotaped 
using the Chelsea Bank program consisted of eight strategies, which are defined on the next four 
pages. 
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The analyses of the 50 videotaped segments were recorded on customized coding sheets. 

An individual sheet was used for each segment to code the analysis of the four major 
categories and the eight problem-solving strategies. A sample of the sheet follows: 

Tape Number Time Segment Began Time Segment Ended 

Names of Students Scenario Number 

Teacher Involvement 

Brief description of the segment. Include any of the following key words if they are appropriate to the 
segment: discussion, reading, writing, mathematics, discussion, computer, collaboration. 




Four General Categories 



Basic Skills 


Collaboration 


Problem Solving 


Concepts 











0 = No activity on this category during this segment 

1 = Limited activity but is not a major emphasis 

2 = Major activity, but it may not be intensive all they way through the segment 

3 = Major activity throughout the entire segment 



Eight Problem-Solving Descriptors: Analysis of Problem-Solving Activities 



Category 


Rating 


Notes 


Focused on Problem 






Previous Specific Learning 






Previous General Learning 






Make Predictions 






Collaborate in Problem Analysis 






Recognize Conjusion 






Make Changes 






Seek Help 







0 = No activity on this category during this segment 

1 = Limited activity but is not a major emphasis 

2 = Major activity, but it may not be intensive all they way through the segment 

3 = Major activity throughout the entire segment 
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At this point, the raters divided the 50 segments equally among themselves. Segments 
were then viewed by two observers (independently) at all times. Each rater had approximately 33 
segments to analyze. Each rater reviewed half of his or her segments in common with one of the 
other raters and the remaining segments in common with the other of the three raters. All ratings 
were conducted independently, and comparison and discussion sessions were held after the 
completion of the ratings. 

Each rater had to observe and analyze the segments by coding the general categories and 
problem-solving strategies. Moreover, the observers had to discuss any segment for which there 
was a coding difference of 2 or higher (based on a scale of 0-S) between their two ratings. (The 
rating form is provided above.) Agreement to within 1 on the rating scale was attained by 
viewing the segment together and discussing their discrepancies until agreement had taken place. 
The initial reliability (consistency) of the ratings prior to any discussion are discussed below. 
Teacher involvement was added as a category to the analysis. 

Although these segments focus on the students' interactions, teachers' involvement is also 
analyzed. Throughout these tapes one can listen and view how the teacher interacts with the 
learners. Therefore, a category was added in order to describe teacher involvement. The amount 
of involvement is categorized between a 0 (no involvement) to a 5 (answers/asks many questions 
and directs almost the entire segment). 

There was high reliability of the ratings across coders. 

Table 4.3 and Figure 4.1 provide data regarding the consistency of the coding for each of 
the segments. The agreements in Table 4.3 are organized by the three major category groups that 
were analyzed. The first is the degree to which the teacher was involved. The second group 
contains the four general categories. The third provides data on the eight problem-solving 
strategies that were observed. 

Each segment was coded by two different coders. The segments were assigned to the 
coders so that each of the coders rated a common set of segments with each of the other two 
raters. The consistency of the coding indicates that across the 50 segments and 650 categories (13 
X 50) the coders were in exact agreement 82% of the time, and over 99% of the time the ratings 
differed by a point or less. 

Some of the differences in ratings are attributed to the length of the segments, which 
ranged fi-om about a minute to twenty-five minutes. The average segment length was 1 minute 17 
seconds, and the standard deviation was about 1 minute. During these segments, a number of 
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different things were occurring which led to some inconsistent ratings. However, the team felt it 
was important to analyze segments that had a holistic integrity as the students dealt with a 
particular problem. The raters also rated segments more than once in the initial stages of the 
process to check on their own consistency of ratings. For instance, the first five segments were 
reviewed at least a dozen times and the next five about five times by each of the raters. 

In the beginning, the inconsistencies in the ratings were significant. However, with 
clarification, discussion, and revisions of categories, the rating consistency increased. The raters 
attempted to arrive at 100 percent agreement with these initial segments before moving to the 
larger database. The agreements and disagreements in the ratings were the basis for extensive 
conversations about what was observed and not observed, and this led to clearer understanding of 
the meanings of the various categories. 



Table 4.3 Agreement of Coding 

by Three Viewers for 650 Video Segments (%) 


Category 


Exact 

Score 


Within 
One Point 
(+/-) 


Within Two 
Points (+/-) 


Teacher Involvement 


45 (90%) 


5 (10%) 


0 


Four General Categories 


1. Basic Skills 


44 (88%) 


6 (12%) 


0 


2. Collaboration 


46 (92%) 


4 (8%) 


0 


3. Problem Solving 


39 (78%) 


1 1 (22%) 


0 


4. Concepts 


38 (76%) 


1 1 (22%) 


1 (2%) 


Eight Problem-Solving Strategies 


1. Focused on Problem 


46 (92%) 


4 (8%) 


0 


2. Previous Specific Learning 


31 (62%) 


19 (38%) 


0 


3. Previous General Learning 


42 (84%) 


7 (14%) 


1 (2%) 


4. Making Predictions 


43 (86%) 


7 (14%) 


0 


5. Collaboration in Problem Analysis 


40 (80%) 


10 (20%) 


0 


6. Recognize Confusion 


36 (72%) 


14 (28%) 


0 


7. Make Changes 


44 (88%) 


6 (12%) 


0 


8. Seek Help 


38 (76%) 


9(18%) 


3 (6%) 


Total 


532 

(82%) 


113 

(17%) 


5 

(1%) 



The lowest agreement was in the category of Previous Specific Learning. This category 
was defined as whether or not students were using knowledge, skills, and information they had 
acquired during previous Chelsea Bank scenarios. Many of the disagreements arose over the use 
of computer skills. Some of the raters were sure the students were using computer skills they had 
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acquired previously while others felt the skills were general skills the students brought with them 
to the task. A second area that caused the coders to disagree was that of Recognize Confusion. 
The category was defined as instances where the students indicated that they knew they were 
confused and did not know what to do next. Sometimes the coders felt the students were 
confused, but they knew where to turn for help. Other times the coders felt the students were 
confused, but they did not recognize that they were on the wrong track. 

Despite the limited range of the scale, the ratings did seem to be remarkably consistent 
across the 50 segments. Determining the reliability of the ratings engendered among the team a 
feeling that they imderstood distinctions across the categories they were looking for and that they 
indeed had a reliable system of describing what the students were engaged in as they completed 
the Chelsea Bank activities. 



Figure 4.1 : Percent of Rater Agreement 
for 12 Categories from the Video Segments 




Exact +/-One +/-Two +/- Three 

Agreement Score Scores Scores 



What did we find out about student behavior during the scenarios? 

An analysis of student interaction during the 50 segments identified while the students 
worked on the Chelsea Bank scenarios revealed that they were engaged and focused on the task 
at hand. Student behavior during the segments was rated on four general categories: 

1 . How much Use of Basic Skills, such as reading, writing, and mathematics, took place in 
the segment as the students worked to solve the problem presented? 

2. How much Collaboration was there as the students worked on the problem? 

3. To what extent did the students concentrate on Problem Solving? 

4. Was there Use of Banking Concepts as the students worked? 

All four of these categories were rated 2 or above on a three-point scale, and comments recorded 
during interviews with teachers and students using the program appeared to confirm that the 
participants considered these considerations an important result of the program as well. 
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Overall, the students utilized a variety of basic skills; they were indeed focused on 
solving the problems presented; and they demonstrated that they had learned and were 
continuing to learn a variety of basic concepts related to banking. The degree to which 
collaboration was observed is discussed below. All of these conclusions can be derived from an 
examination of the ratings of the four general categories included in Table 4.4 and Figure 4.2. In 
addition, samples of teacher and student comments that reinforce the observations of the viewers 
are included in this report. 

How prevalent was behavior related to the four general categories? 

The ratings of the four general categories were determined by judging holistically what 
was seen during the entire segment. As already detailed, each segment was rated by two viewers 
and their ratings were averaged; and as the reliability already reported shows, there was very 
high agreement between reviewers across segments. The behaviors that were the focus of the 
four general categories may not have been occurring continuously, so the judgments were based 
on the viewers’ analysis as to how strong or weak these behaviors were throughout the segment. 
A score of 3 indicates the judgment that the behavior was strongly exhibited, while a score of 0 
indicates that the behavior was not seen at all. 

Use of Basic Skills . The Use of Basic Skills category was rated highest of the four general 
categories (2.76), and the standard deviation for this category was the smallest (.51). The 
students were reading, writing, and using basic math skills throughout the segments. The raters 
varied little in their judgment that basic skills were seen consistently and strongly throughout all 
of the segments. This supports one of the main emphases of the Classroom, Inc. goals to get 
students to develop their basic skills through use in practical applications. 

While teachers welcome the amount of mathematics the program involves, many feel that 
it could require more reading and writing: 

• “. . .a weakness of the program is that it does not involve enough reading and writing; 
[it could require students to take] notes to inform decision making.” 

• “I like Chelsea. . .because there is more math. . ..It focuses on money and thinking 
mathematically.” 

At the same time, several teachers commented on the demands of the reading and writing 
required in the program: 

• “There were no other problems in reading the information off the screen.” 

• “Sometimes the language is too difficult. Some of the terms. . .are not familiar to 
them.” 

• “Sometimes the students had difficulty putting their decisions. . .into words. They 
seemed to know what the consequences were, but they had difficulty writing what 
they thought.” 
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The students who were interviewed clearly thought there was a great deal of reading and 
writing: 

• “Reading — there’s a lot. . . .you have to read everything. . . .they don’t talk to you. If 
you don’t read, you don’t know what is going on. [The] reading [is] not hard — in 
[the] manual or glossary, and you can access that any time. Writing — ^yes, in [the] 
bank [setting], no, in [the] hotel. [In the] bank — you have to write for it to be accepted 
as [the] answer.” 

• “Lots of reading. . ..you have to read fast — lots of pressure to read fast. I want to stop 
the time sometimes.” 

• “Need to write a report about why I made the decision. Samuel did the writing — had 
to write what the problem was.” 

• “[We are] writing [because] we’d jot everything down.” 

• “[We’ve] got to read what they say. [And we] go to [the] manual, and it tells you 
what to do.” 

• “Michelle usually reads it aloud; [and it’s] not too hard. We can read the words but 
sometimes [the] sentences are hard to imderstand.” 

• “We had [the] computer to give help, but a paper manual would have been good.” 

Although the students were able to cite the reading and writing they felt the program 
requires, some felt, as did the teachers, that the program involves mathematics more: 

• “You look at [a] check....youleamaboutchecking, math. You are a teller.... the math 
is the most memorable because it is the thing you think about the most. 

• “. . .basic math helped.” 

Problem Solving . It is also clear that along with using basic skills, the students were 
engaged in Problem Solving. The viewers felt that there was seldom a time when they were 
distracted from the focus of the scenario. They were engaged in searching for an answer and 
determining what they were to do next. This does not mean that they were involved in all aspects 
of problem solving as that term has been defined by a variety of researchers and theorists, but it 
does mean that they followed the scenario problem and searched for a response to the situation. 
The rating of 2.24 for Problem Solving, and the modest standard deviation for this score, 
indicates little variance in this highly ranked behavior. 

Both the students and the teachers emphasized problem solving as perhaps the most 
prominent requirement of the program. The students’ comments about these demands reflect a 
genuine enthusiasm: 

• “Decision making was the hardest thing — ^kept getting fired. Why? Decision making 
is hard because you want to be right.” 

• “We learned to work it out, [even if we were] not always right; [we learned to] look at 
other points of view.” 
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• “There’s thinking. . .about the problem; think what to do next — ^why [a] decision 
affects you, the rules to follow, and about the customer.” 

• “We had to evaluate to make [a] decision; there was nothing in the manual guide for 
this problem.” 

• “If you don’t think about it, you don’t get it.” 

• One student emphasized how good decisions depend on good information. 

• “It keeps your brain going — in school, out of school.” 

• ''’‘Chelsea is a lot of thinking.” 

Just a few samples show how their teachers were equally enthusiastic about this aspect of 
the program included these: 

• “The most important skills they are learning are critical thinking, problem solving, 
and professionalism.” 

• “They had to think about the consequences. It was good for them to think about that 
problem.” 

• “They make decisions based on the problems. It goes beyond the surface — why do 
this? Regular class [activities don’t] always do this. [Giving this kind of] education in 
books is one way; [these] simulations [come] closer to reality, real experience.” 

• “[The] kids are. . .learning. . .decision making, teamwork, negotiation.” 

• “[Among] the most important things the students learned were. . .imderstanding 
options and feedback — ^they want feedback.” 

Use of Banking Concepts . The rating of Use of Banking Concepts (2.19) indicates that as 
the students worked on the Chelsea Bank scenarios, this behavior was taking place to a 
significant degree. The range of scores as indicated by the standard deviation indicates that there 
were times that the use of banking concepts was not very strong, but this would be expected as 
students could not be expected to use banking concepts all of the time. 

Teachers felt, as one said, that learning new concepts such as banking and money was 
almost as important a result of the program as practicing basic skills and problem solving. Other 
comments about this category included: 

• “Helps kids to understand jobs, areas for the future.” 

• “They are. . .learning basic concepts about balancing checkbooks, counting money, 
etc.” 

• “Some of the terminology was difficult, but I tried to cover that in business class.” 

Student comments during interviews often revealed a newly developed interest in 
banking and banking concepts, but most often students commented on learning skills relevant, 
but not exclusive, to banking, such as dealing with people and working with computers: 

• “[I] learned how to work as a teller. . ..Each customer has a new problem.” 
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• “[We] talked about several of the types of tasks including check cashing, opening 
accounts, etc.” This student found that a bank teller’s job is not boring. 

• “I now know more about banking when I go to the bank with my mother.” 

• One student stressed a point made by numerous others: learning to deal with people. 
He felt he learned how to put up with people and to treat people appropriately, how to 
manage money, generally what a bank teller does, and computer keyboard skills. 

• Another student emphasized the value of learning about management in the business 
world. 

• “Computer skills and technical problems with the computer were not big 
problems.... We learned a lot.” 

Collaboration . The general category Collaboration was rated 2.16 overall. In rating this 
category, the team was looking for the degree to which students worked together. For example, 
when one student handled the mouse and another entered information from the keyboard, while a 
third watched and commented on their progress, a viewer would record evidence of 
collaboration. But the students sometimes worked independently even though they were all 
seated in front of the computer and were working on the same scenario. They seemed to be 
working independently in parallel fashion rather than cooperatively. Many times one student 
would come up with a response to some detail of the scenario and ask the others what they 
thought of the solution, and they would usually agree with the response with little discussion, 
debate, or modification of the suggested response. 

Teachers and students emphasized the importance of collaboration so continuously in 
their assessment of their experience with the program that this emphasis must be considered 
when noting that while the viewers rated collaboration high, it was the lowest of the four main 
categories. 

The students discussed both their success in working together to solve the problems 
presented in the program and problems they encountered working together. For example, they 
noted that collaboration was limited by team members who were absent, by some teachers’ 
control over the role played by particular members of a group, by reticent team members, by 
fellow students who tended to exercise control over the computer, etc. Frustration with other 
team members appeared strongest among students who were inclined to assume leadership roles: 

• “I could do it by myself When it’s two against one, you have to choose, [and] you 
can be wrong. I felt good being the leader.” 

• “My group had three people. They bother me too much; they talk too much.. . .We 
need to work as a team to make decisions, to take turns — on [the] mouse, the 
keyboard,... [reading] the manual out loud.” 

• “I liked the group work; they helped me if I had a problem; our group got along.” 
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• “We learned to work it out — ^not always right. [We learned to] look at other points of 
view. [The] most important thing [was]. . .working together.” 

• “In our group we cooperated; we switched every session. We just did the switching. 
Sometimes you get along; sometimes you don’t. [Things work out] if you get on their 
tails; [you may] have to push others to get [your] grade.” 

• Sometimes you don’t get along with people but you have to leam to understand others 
[and] get along.” 

Clearly, it was the structuring of teams that made teamwork a necessity and created the 
opportunity for, and importance of, collaboration. Many comments from the teachers echoed 
this: 

• “They need the group work! I let them choose their own partners.” 

• “Working cooperatively. . .they understand relationships. One student at each 
computer wouldn’t work. They have to leam to take different roles and work with 
other students.” 

• “I think they are learning to work with others. They talk. . .and work through their 
differences. Sometimes they get upset.” 

• “The team work is critical. In a group, a poor reader. . .may be a great calculator, [and 
the team approach gives] him or her a chance to participate.” 

Thus the teachers invariably stressed the program’s potential to get students to respect 
each other, to listen to each other, to compromise, and to collaborate — to collaborate effectively. 

In defining problems, as well as in developing responses, there was not a great deal of 
collaboration noted by the viewers, however, as will be shown in the examination of the eight 
problem-solving strategies. 

What can be concluded about the four general categories? The overwhelming conclusion 
is that the students were focused on solving the scenario problems and they applied a variety of 
basic skills. They used banking concepts as they went about their tasks and to a slightly lesser 
extent they collaborated in their work. However, the collaboration could be characterized as 
“cooperation” and “offering mutual support” — as opposed to interaction that tmly reflects 
collaboration in developing a product. 



Table 4.4 Ratings of the Four General Categories 


Categories 


Mean Rating 


Standard Deviation 


1. Use of Basic Skills 


2.76 


.51 


2. Collaboration 


2.16 


1.02 


3. Problem Solving 


2.24 


.88 


4. Use of Banking Concepts 


2.19 


.91 
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Figure 4.2 Ratings of Four General Categories 




Were the students using the eight problem-solving strategies? 

The area of Problem Solving was characterized by eight strategies based on a review of 
the literature on problem solving as well as on an analysis of what was seen during the fifty 
segments. Were the students engaged in problem solving? Or were they merely going through a 
series of activities which they felt were parts of the procedure required to determine a correct 
answer? Is this just a linear program that focuses on following prescribed steps while calling for 
little thinking — or is it a program which emphasizes independent thinking in identifying 
problems, conjecturing about solutions, and modifying responses to fit each situation? 

The analysis we conducted suggests that the program is linear and that students are 
pushed to find the “best answer.” However, they are also led to determine what the problems are, 
to learn to use resources, and to consider the consequences of their responses. 

The eight Problem-Solving Strategies examined during the segments were: 

1 . How Focused on the Problem were the students during the segment? 

2. How often did students use Previous Specific Learning gained fi'om the Chelsea Bank 

program? 

3. What evidence is there of students pulling Previous General Learning into their problem- 

4. Do the students Make Predictions while solving the problems? 

5. How much Collaboration in Problem Analysis takes place? 

6. Do the students Recognize Confusion when they are indeed confused? 

7. Do the students Make Changes in their thinking as they solve the problems? 

8. Do the students Seek Help when they need it? 
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In discussing their experience with and reactions to the program during interviews, 
teachers and students had more to say about a few of these strategies than about several others. 
Their comments are included to help explain the ratings based on what the viewers saw. 

Focus on the problem . The Problem-Solving Strategy that was rated the highest was that 
of being Focused on the Problem. This was expected because each scenario was structured to 
present the students with a problem to be solved. However, the students did more than accept the 
problem. There was considerable discussion of the nature of the problem and about what they 
were expected to do in response to the problem. 

Had the problems been more open-ended, it is possible that they might have promoted 
more analysis, but one wonders if this might instead have led to the students becoming lost while 
trying to determine what they were to do. The high rating (2.94) for this strategy and the 
relatively small standard deviation indicates that the students focused on the problem and that 
they were constantly discussing or mentioning the problem. 

Many of the comments from teachers and students about this general category indicate 
quite clearly how intent the students were on defining and solving the problem. Two additional 
quotes — ^the first from a student and the second from a teacher — are typical of what the 
interviews revealed about the focus on the scenario problems in these classrooms: 

• “.. .learn the problem; go to the problem; figure it out; go on to the next customer. We 
talked it out in our group and then [would] do over what we had to do.” 

• “[They] look at [a] problem from global and specific perspectives. [They learn how 
to] make an appropriate decision and to be able to justify that decision. It encourages 
metacognition.” 

Collaborating and usins the Chelsea experience . Two problem-solving strategies which 
were operating across the segments at a moderately high level were Collaboration in Problem 
Analysis and the use oi Previous Specific Learning. It should be noted that the strategy of using 
previous specific learning was rated only in response to information and knowledge students had 
gained from previous Chelsea Bank scenarios. For example, the use of the various tools was 
evident as was other information gained from previous scenarios. 

These two categories were each rated at about 2.0, indicating that the students often 
talked with one another about the nature of the problem in which they were engaged and how 
they might solve it. In doing so, they used previous information they had learned in other 
scenarios. These two categories give strong evidence of students’ willingness to share their 
imderstanding and analysis of a problem and their ability to gain knowledge and then apply it in 
subsequent situations. 
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Several comments from teachers are typical of how the teachers’ enthusiasm for the 
H program’s demand for collaboration was frequently specified as that needed for problem solving: 

*' • “I think that what they learned most from the simulations was teamwork (problem 

solving as a group). Next in importance was sharing decisions.” 

■ • “I don’t think the simulations would be as effective if the students worked alone. 

They wouldn’t have anyone to interact with and to leam from. They need to check 
M with a peer and discuss their choices.” 

* Student reports verify what the teachers noted about collaborative problem solving: 

gk • “If we didn’t agree, we’d work it out.” 

■ • “We talked, argued about [the] decision. We made the wrong decision one time and 

we felt bad — ^the boss yelled at us.” 

'■ • “[We had to] make [a] decision. He said no. [I] asked why. We negotiated the 

" decision.” 

■ The students’ comments about how they learned about banking and the Chelsea Bank in 

“ particular may seem obvious, but they illustrate their awareness of how they were applying — or 

M would have liked to apply — ^previous specific learning: 

V • “[I relied on the] Chelsea manager’s guide book. I liked the investigation aspect — 

looking for information. That also makes it fun and interesting. At [the] bank, [I] can 
H go back and run a check on them.” 

* • “What’s hard [is when I] don’t know what to do; [I] have to go to [the] manual, and 

^ sometimes the answer is not there.” 

^ • “It got easier to count money.” 

• “. . .our group did get more comfortable with handling the problems.” This student felt 

H that some students got jobs because of the simulations. 

Some teachers — ^but not all— stressed the importance of learning and applying this specific 
A experience gained from the program: 

■ • Several noted that assessments designed for the program should measure to some 

extent how much the students have grasped about banking. Another teacher, however, 

■ specifically argued that the assessment should not cover banking procedures, but 

* rather general skills learned, such as cooperation. (Clearly, the assessment will need 

^ to involve the grasp of previous specific knowledge as it serves subsequent problem 

■i solving.) 

• The Chelsea program was attractive to some teachers because what could be acquired 

jji from it about banking and then applied required the use of mathematics. 

“ • One teacher’s implication was clearly that the practical information is very important: 

“I’d like to move beyond simulations to job experience.” 

H Recognizing confusion and seekins help . Factors operating in both Collaboration in 

Problem Analysis and use of Previous Specific Learning were the students’ ability to Recognize 
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Confusion — crated .89 — and their willingness to Seek Help — crated 1.07. The first of these 
involved knowing that they needed help, and the second included being aware of where to look 
for help. While these strategies may seem, on the smface, to be rather low-level aspects of 
problem solving, they are defined by some researchers and theorists as the epitome of problem 
solving. To know when you don’t know and to seek the help you need, they argue, is the highest 
level of problem-solving behavior. 

The viewers saw only moderate evidence that the students were engaged in these 
activities, however. There were times when teams were unsure about what to do next, but these 
times were often related to the mechanical aspects of completing a scenario rather than an 
understanding of what was confusing about a problem. Comments were often of the nature of 
“What do we do next?” rather than, “I can’t understand why the situation can’t be solved this 
way?” The search for help was focused almost entirely on the use of the tools provided with the 
scenario or a raised hand requesting teacher aid. 

It would have been very encouraging if students had on occasion conjechxred about a 
problem, brainstormed about what was known about the problem, and then developed alternative 
solutions to the problem. This kind of thoughtful understanding of problems and sharing and 
conjecturing about solutions was not often seen. 

Several teachers accounted in the interviews how students tended to call to them for 
guidance when the students recognized that they were confused, and a few others emphasized 
that they reminded students about the help tools provided in the program. A few students, too, 
confirmed what the teachers said about confusion leading to a call to the teacher. Several 
mentioned the manual, guides, and rules as sources of information that could resolve questions 
arising during problem solving. 

One possible reason for the low scores on these two problem-solving strategies is that the 
program appears to put considerable emphasis on negotiation. Studying student comments about 
collaboration, for example, suggests that disagreement among members of a team about what to 
do might be considered recognition of confusion. Such differences of opinions needed to be 
resolved, if possible. One way a few students noted doing that was reviewing the information 
presented in the scenario. If what these students report was going on, the viewers may not have 
noted it as recognition of confusion. They did note calling for the teacher and consulting the 
guide book and other program features as seeking help. 

• “When the students were having problems figuring out what to do, they would call 
me over for help and to ask questions. Arguing was an indication that there was a 
problem. Their voices got louder and the argued about the responses.” 
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• “The biggest problems students had was when they couldn’t decide who was right. I 
would urge them to go back and use the information sources and think their way 
through the problems.” 

• One student noted that the teacher intervened only when the group had made all the 
wrong decisions. 

• “. . .if we don’t get it, ask the teacher.” 

Seldom-used strategies . The aspects of problem-solving behavior that were scored lowest 
across the 50 segments were those of using Previous General Learning (.39), the inclination to 
Make Predictions (.39), and the decision to Make Changes (.18). The standard deviations for 
these three strategies are large enough to indicate some variance in the frequency of their 
occurrence. However, the low scores indicate they were not observed very often. 

The infrequent use of previous general learning was perhaps the most surprising. We had 
expected students to apply ideas, knowledge, and perhaps even problem-solving strategies they 
had learned elsewhere. However, such was not often the case. The extensive research on the 
limitations of transfer of learning should, perhaps, have made us less optimistic about this. The 
students seemed to treat the Chelsea Bank experience as a separate, albeit interesting and 
engaging, activity. They did not appear to associate it with other school activities and projects or 
with those that occurred outside of school. 

One very strong probability for the lack of use of general information is that many 
students in these classes had very limited knowledge gained from the experience of going to 
banks. There were indications from the interviews that substantiated this. Thus knowledge about 
banks and banking was not available in their backgrounds. Also related to the low use of 
previous general knowledge is the probability that the students would not in a normal discussion 
verbalize their prior knowledge. This would not mean that they would not be using it 
necessarily — only that it could not be observed. It is easy to see what they are saying and doing 
to solve the problem at hand, but their use of prior knowledge is far less obvious. 

In the light of the low rating for this strategy, teacher comments from the interviews — 
such as these examples — about transfer of learning become quite revealing and important, we 
think: 

• “Many of [the students] made connections to home concepts, such as balancing the 
checkbook. They said they now feel more comfortable in a bank and they know more 
about what a bank is all about.” This teacher also reported that the class had to lay out 
a bank in a class exercise — showing where all of the different components would be. 

• “The bank is more familiar to them, and the hotel is not so familiar. There is more 
immediate application for Chelsea. 
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• “The math teacher is doing some things with Chelsea Bank in math class. I’m not 
sure how we could get the integration with other subjects going, but I think [it] could 
be done....” 

• “There was not much communication with other teachers in the schools about the 
program.” 

• “The reality orientation was important, but there are some students who had not been 
to a bank. We had to discuss why the various incidents might have happened. For the 
project, we would spend 20 minutes discussing a previous scenario.” 

Student comments indicate that a transfer of general learning was going on but not 
necessarily observable: 

• One boy who was working as a department manager in a large drugstore in downtown 
Brooklyn said he sees similar problems: the need not to lose one’s temper with 
customers, the need to be patient, the importance of listening. 

• One girl who was working in a bank noted to her teacher that there was more reading 
and writing in the real-world bank than in the simulation. 

• A student reported how his math teacher had been willing to field questions that grew 
out of the Chelsea Bank experience. 

• A student noted how there is — as at Chelsea — often a line outside real banks. 

• One student reminded us that transfer works both ways but said that nothing from the 
classroom experience helped outside of school. 

• “Out of school, [we need to] learn patience; I understood [the] stress. . .clerks [face].” 

• “Personalities play an important part in real life, too.” 

Perhaps the lack of any making of predictions or changes was most closely related to the 
structure of the scenarios. While the students were to consider consequences, they were seldom 
asked to make any other predictions about the problem or possible solutions. They went through 
the scenarios in a step-by-step fashion, focusing on, “What do we do next?” Since there were few 
predictions and only limited conjecturing about solutions, it is not surprising that there was little 
evidence of modifying predictions or answers. The students would select answers, and to some 
extent that was a prediction. However, it was taken from a list of alternatives. When the students 
had the wrong answer, they were not expected to develop a different response. 

Several student and teacher comments reflect on making predictions and changes: 

• One boy expressed approval of the program because it made him think about the 
decisions he had to make — ^that the manager’s guide didn’t give all the answers. 

• One student noted what several others implied: that fear of making the wrong 
decision became a kind of incentive for considering consequences: One bad decision 
affected everything. 

• “. . .you had to make hard choices. [We] need[ed] to use common sense. . .as a group 
and come up with a good choice.” 
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• Another student suggested how important prediction was in the program: . .always 

investigate before making a decision; get someone else’s opinion; don’t act [as if] 
you’re always right; learn how to weigh alternatives; compromise.” 

In the light of how seldom viewers identified this strategy, however, one teacher’s 
comment becomes highly significant: “Some students always wanted to skip steps and just kind 
of guess what the best response was.” 

Many of the student comments about problem solving and about collaboration indicated 
either that they felt they had made adjustments and changes or that they wished that the program 
had allowed more of an opportunity to do that: 

• One student emphasized how important it was to explain and review decisions and to 
understand how they affect others. 

• One aspect of the simulation that one boy did not like was that he wanted to go back 
and change decisions. He mentioned the problem with the vacation request from the 
cleaning lady. He said that he would like to change that decision. He said that when 
you have better information, you can make better decisions. 

A gain, a teacher’s comment is highly indicative of what was happening — change making 
was a part of the process that preceded decision making. Once the latter was done, the die was 
cast: “The biggest problems students had was when they couldn’t decide who was right. I urge 
them to go back and use the information sources and think their way through the problems.” 

In summary, were the students engaged in problem solving? The answer is a clear 
Somewhat] Given the structure of the scenarios, there were some aspects of problem solving that 
were limited. On the other hand, students did not show a lot of overt evidence of thinking about 
the problems to a great extent and they did build their understanding and skills from one scenario 
to the next. What they did not do to any great extent was to conjecture and predict responses, to 
make cormections with knowledge and skills beyond Chelsea Bank, or to learn to fix up and 
change responses so they would be more appropriate, effective, or clear. 



Table 4.5 Ratings of the Eight Problem-Solving Strategies 


Strategies 


Mean Rating 


Standard Deviation 


1. Focused on Problem 


2.94 


.19 


2. Previous Specific Learning 


2.05 


.79 


3. Previous General Learning 


.39 


.62 


4. Make Predictions 


.39 


.87 


5. Collaboration in Problem Analysis 


1.96 


.99 


6. Recognize Confusion 


.89 


1.03 


7. Make Changes 


.18 


.40 


8. Seek Help 


1.07 


1.07 
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Figure 4.3 Ratings of Eight Problem-Soiving Strategies 
(50 segments) 




How did teacher presence affect student problem-solving behavior? 

A major factor that seemed to affect the students’ behaviors was the presence or absence 
of the teacher during the time the students were working. The team felt that this factor 
significantly affected the students’ behaviors. It was, therefore, decided to rate the extent to 
which the teacher was present and involved during the duration of each of the segments. 

The percent of each of the four ratings are shown in Figure 4.7. The ratings ranged from 
not involved or none (0), to extensive involvement (3). It was decided that for analysis, we would 
combine those segments for which the teacher received either a 0 (not involved) or a / (limited 
involvement). In segments rated 0 the teacher was not seen at all. In segments rated 1 the teacher 
was seen, but for only a short time; the teacher may have made a passing comment, but s/he did 
not become involved in the lesson to any extent. 

We then decided to compare the combined 0 and 1 segments to those segments in which 
the teacher was rated as extensively involved (3). This meant that we were emphasizing the 
differences, since we were not analyzing those segments in which the teacher was somewhat 
involved (2) at all. We felt that in comparing the segments rated 0/1 to those rated 3, we would 
emphasize any effect the teacher may have had. 

There were a total of 27 segments in which the teacher rating was 0 ox 1 and a total of 9 
segments in which the teacher rating was 3. Thus the total numbers of segments in each category 
was a ratio of 3: 1 . However, the difference in length — ^the total amount of time consumed by the 
segments in each rating category — ^was not quite as large. The 9 segments for which the teacher 
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rating was 3 covered a total of 58 minutes (almost one hour) while the 27 segments that were 
rated 0 ox 1 covered a total of 144 minutes (two hours and 24 minutes). 

Table 4.6 and Figure 4.4 show the differences in the four general categories with the 
teacher present and the teacher not present. When the teacher is present, the students are often 
involved in a question/answer dialogue with the teacher and, as a result, the basic skills activity 
and the student collaboration decline. The differences in the problem solving and the banking 
concepts categories are small, suggesting that teacher presence has little impact. This is probably 
because the students are focused on the problem at hand whether the teacher is present or not and 
they use banking concepts in either case. 

A similar pattern is revealed for the eight problem-solving strategies, shown in Table 4.6 
and Figures 4.5 and 4.6. When the teacher is extensively involved, there is a significant decline 
in collaboration in problem analysis. On the other hand, the strategies of seeking help and 
recognizing confusion increase significantly. When the teacher is present, s/he is often there at 
the behest of the students, and/or the teacher has recognized the students are having difficulty. 
Figure 4.7 shows the overall percentage of ratings for teacher involvement. 



Table 4.6 Ratings of the Four General Categories and Eight Problem-Solving 
Strategies For High and Low Teacher Involvement 

(High 9 Segments, Low 27 Segments) 


Mean Ratings Standard Deviations 


Categories 


High 


Low 


High 


Low 


1. Basic Skills 


2.39 


2.87 


0.82 


0.32 


2. Collaboration 


1.17 


2.26 


1.11 


0.95 


3. Problem Solving 


2.22 


2.13 


0.97 


0.91 


4. Concepts 


2.0 


2.13 


0.75 


1.05 


1. Focused on Problem 


2.89 


2.94 


0.22 


0.21 


2. Previous Specific Learning 


1.83 


2.06 


1.06 


0.73 


3. Previous General Knowledge 


0.50 


0.33 


0.70 


0.60 


4. Make Predictions 


0.44 


0.31 


0.92 


0.83 


5. Collaboration in Problem Analysis 


1.06 


2.17 


0.95 


0.96 


6. Recognize Confusion 


1.44 


0.50 


1.07 


0.88 


7. Make Changes 


0.06 


0.13 


0.17 


0.42 


8. Seek Help 


1.33 


0.81 


1.19 


1.06 
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Figure 4.4 Ratings of Four General Categories 
for Low and High Teacher Involvement 
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Ratings of Four of Eight Problem-Solving Strategies 
for Low and High Teacher Involvement 
(36 segments) 
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Figure 4.6 Ratings of Four of Eight Problem-Solving 
Strategies for Low and High Teacher Involvement 
(36 segments) 
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Figure 4.7 Percentage of Ratings for Teacher 
Involvement (50 segments) 
(Mean=1.21, S.D.=1.16) 



3=Extensive 

20 % 



2=Some 

24% 



1=Limited 

15% 




0=None 

41% 



What is the nature of the differences between high school students and junior 
high students as they do the Chelsea Bank scenarios? 

A comparison between the junior high and high school students viewed and rated shows 
only a few distinctions of much interest. Junior high students were rated slightly higher on three 
of the four general categories, but high school students rated higher on the use of basic skills. 
(See Table 4.7 and Figure 4.8.) Little can be derived from this analysis, which may suggest that 
the younger students were a bit more flexible when solving the problems, but the older students 
were more secure about performing the reading, writing, and math necessary to do so. 
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Table 4.7 Ratings of the Four General Categories and Eight Problem-Solving Strategies 
Comparing High School and Junior High Students 

(50 segments) 


Categories 


High School 


Junior High 


1. Basic Skills 


2.82 


2.67 


2. Collaboration 


2.1 


2.17 


3. Problem Solving 


2.1 


2.4 


4. Concepts 


2.13 


2.27 


1 . Focused on Problem 


2.92 


2.95 


2. Previous Specific Learning 


2.07 


2.02 


3. Previous General Learning 


0.55 


0.15 


4. Make Predictions 


0.22 


0.65 


5. Collaboration 


1.93 


2.0 


6. Recognize Confusion 


0.68 


1.2 


7. Make Changes 


0.18 


0.17 


8. Seek Help 


1.0 


1.17 
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Figure 4.8 Ratings of Four General Categories 
Comparing High School and Junior High Students 
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As with their inclination to collaborate, learn banking concepts, and focus on the 
problem, the junior high students tended to practice the problem-solving strategies a bit more 
than the older students (see Table 4.7 and Figures 4.9 and 4.10). They seemed more ready to 
recognize when they were confused perhaps because they were less confident in what they were 
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doing and less inhibited about saying, “I don’t understand.” This may also accovmt for their 
seeking help more often. The junior high students’ stronger tendency to make predictions may be 
because of their uncertainty about correct responses and their willingness to articulate possible 
outcomes. 

The high school students were clearly more able to apply previous general learning — an 
understandable result, since they would surely have more experiences to bring to the problem- 
solving procedures. They were slightly more ready to apply what they had learned from the 
program (specific learning). 

All of these conjectures about differences between high school and junior high students 
are attempts to explain differences that are, in fact, rather minimal. The major conclusion that can 
be drawn about this comparison is that the two groups seemed to exhibit very similar behaviors 
as they worked on the Chelsea Bank scenarios. 
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Figure 4.9 Ratings of Four of Eight Problem-Solving Strategies 
2 95 Comparing High School and Junior High Students 

(50 segments) 
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Figure 4.10 Ratings of Four of Eight Problem-Solving Strategies 
Comparing High School and Junior High Students 
(50 segments) 
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What are the differences in student behavior between the first Chelsea Bank 
scenario and a later scenario? 

One of the questions that arose during the analysis of the videotape segments was 
whether there were student behavior differences in different scenarios. We were confident that 
we did not have adequate video segments for every scenario to do a comparison across all fifteen 
scenarios. In fact, we did not have any video segments for scenarios 3, 12, 13, and 14. Therefore, 
a comprehensive comparison was not possible. When the segment selection was completed, we 
constructed a table (Table 4.2) to determine which scenarios were represented most often among 
the segments we analyzed. 

Table 4.2 indicates that scenarios 1 and 10 were each represented by 10 segments. In 
addition, the total time for the these two segments was almost 92 minutes for scenario 1 and 
approximately 68 minutes for scenario 10. It seemed reasonable to subject these two scenarios to 
a comparison since there was considerable data for each. 

So in order to consider the effect of time and experience with the program on student 
performance, the ratings of viewers were compared for the first scenario in the Chelsea Bank 
series and scenario 10. The means of the ratings for scenario 1 are higher than those for scenario 
10. In completing scenario 1, the students were considerably more involved in solving the 
problem, relied more on basic skills, appeared more attuned to banking concepts, and were more 
ready to collaborate (see Table 4.8 and Figure 4.1 1). 

Across the ratings for the eight problem-solving strategies, there were only slight 
differences in how focused on the problem the students were, in their inclination to make 



Chelsea Bank Simulation: Problem Solving and Assessment Development 



predictions, and in the use of what was learned from the program. The later is perhaps surprising, 
since by scenario 10 the students had considerably more exposure to banking concepts to apply 
when arriving at a solution — yet they used that information less. Since the ratings for being 
focused were relatively high, the difference may not mean a great deal. It was also slight for 
making predictions, but this is the only strategy where the ratings were higher for the later 
scenario. There were across the time/experience span, moderate differences in the amount of 
collaboration on problem solving, the use of previous general learning, and making changes — all 
with the higher ratings on the earlier scenario (see Table 4.8 and Figures 4. 1 2 and 4. 1 3). 



Table 4.8 Ratings of the Four General and Eight Problem-Solving Strategies 
Comparing Scenario 1 and Scenario 10 

(20 segments) 




Scenario One 


Scenario Ten 


Categories 


Mean 


S.D. 


Mean 


S.D. 


1 . Basic Skills 


2.9 


.31 


2.1 


.74 


2. Collaboration 


1.7 


.95 


1.1 


1.14 


3. Problem Solving 


2.6 


.46 


1.28 


1.15 


4. Concepts 


2.0 


.53 


1.39 


.96 


1. Focused on Problem 


3.0 


0 


2.78 


.36 


2. Previous Specific Learning 


1.7 


.86 


1.5 


.90 


3. Previous General Learning 


.60 


.81 


.22 


.36 


4. Make Predictions 


0 


0 


0.56 


0.17 


5. Collaboration 


1.50 


.94 


.94 


.88 


6. Recognize Confusion 


1.55 


1.55 


.33 


.56 


7. Make Changes 


.25 


.42 


0 


0 


8. Seek Help 


1.70 


1.13 


.17 


.35 



In the first segment, the students were significantly more ready to recognize when they 
were confused and to seek help. This result was definitely to be expected. But overall, what 
might explain the higher means for the earlier scenario? 

• It may be that scenario 1 has more appeal and is more involving that scenario 10. 

• Perhaps by the time that the students have reached scenario 10, they have come to know 
what to do and their involvement takes on some degree of automaticity. The observation 
of the viewers is that the simulations are somewhat linear, and if this is so, it would 
contribute perhaps to a decrease in student commitment and involvement — a drop in 
intensity might be expected. 
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Figure 4.11 Ratings of Four General Categories 
Comparing Scenario 1 and Scenario 10 
(20 segments) 
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• The segments with scenario 1 are shorter in length; thus they may maintain more 
intensity in terms of student involvement. The teacher is more likely to come into the 
segment and to perhaps direct and refocus their behavior. Clearly, there was less teacher 
involvement in scenario 10. 

• The higher problem-solving scores on scenario 1 for both the General Categories and the 
Problem-Solving Strategies analysis may indicate that scenario 1 lends itself more to 
problem solving as the students face the problem of what do with a customer. Perhaps 
there is less problem solving involved in scenario 10 when the students have taken on the 
role of a customer service agent and are concerned primarily with applying bank routines 
and procedures. 

• The higher score on recognizing confusion in scenario 1 may merely be the result of 
confusion with the routines of the Chelsea Bank simulation program and a willingness of 
the students to express this confusion at this early stage. 

• Likewise the lower score on seeking help on scenario 10 may be the result of the students 
“getting the hang of it” and being able to proceed through the simulation with little need 
for assistance. 

• It is also possible that the students were somewhat complacent with the process by 
scenario 10 and their interactions were reduced and they were just pushing to get the 
scenarios completed. The lack of verbalization during the videos would, of course, lower 
the ratings on all categories. 

• It may also be that the groups that were primarily videotaped in scenario 1 were different 
from the groups videotaped for scenario 10. 
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Figure 4.12 Ratings of Four of Eight Problem-Solving Strategies 
3 Comparing Scenario 1 and Scenario 10 

(20 segments) 
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Figure 4.13 Ratings of Four of Eight Problem-Solving Strategies 
Comparing Scenario 1 and Scenario 10 
(20 segments) 
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Four representative segments show the effect of teacher involvement. 

The research team conducted a detailed analysis of 50 segments, which were identified in 
the MindWorks videotapes. The analysis of the student and teacher behaviors in the 50 segments 
constituted the major quantitative analysis of this part of the study. However, these data may not 
adequately describe what was commonly observed across these 50 segments. Therefore, the team 
decided that a set of four segments should be selected to provide a more holistic view of all the 
tape segments that were studied. The narrative below provides a more qualitative analysis. 

The team discussed the selection of the segments and concluded that the single most 
important factor in determining what the students did as they engaged in the Chelsea Bank 
scenarios was whether the teacher was present or absent as the students performed. Therefore, a 
three-phase process that emphasized selection of segments based on teacher involvement was 
followed: 

Phase /: This phase entailed the identification of those segments which were rated with high 
teacher involvement and those with low teacher involvement. 

Phase II: This phase entailed the identification of segments with high ratings for the four 
general categories and eight problem-solving strategies. Thus segments with high 
teacher involvement and high ratings of categories and strategies, and segments 
with low teacher involvement and high ratings of categories and strategies were 
found during this stage. Based on these criteria, eight segments were chosen. 

Phase III: During this phase, the eight segments were viewed and analyzed to determine 

which of them appeared to be the most representative of what occurred in a typical 
Chelsea Bank session. Based on this step in the procedure, four representative 
segments were selected: two segments with high teacher involvement and two 
segments with low teacher involvement. 

Segments #25 and #45 were chosen for several reasons: 

• Teacher involvement was rated low; 

• All twelve ratings, including those for general categories and for problem-solving 
strategies, were high; 

• Two different scenarios (#8 Mr. Smith and #11 Ms. Austin) were addressed; 

• They involved students of different ages (junior high vs. senior high); and 

• In addition, these segments demonstrated typical students' interactions when the 
teacher involvement is low. 

Segments #32 and #40 were chosen based on the high teacher involvement and the 
typical result of that factor: the students are less active in doing the task. 
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Segment 32 


Scenario Number: 8 (Mr. Smith) 


Teacher involvement: 3 


Length: 10.67 minutes 




Scenario Overview: 



This scenario concerns a conflict between a customer and a teller. The customer, who is 
dressed like a bum, wants to cash a check for $4,000. The problem arises when the teller says 
that she is not allowed to cash checks for this amount of money without approval from a 
customer service representative, and the customer complains about the bank service by saying 
that he is insulted. 

Description of Video Segment : 

The teacher truly controlled the students throughout the segment. It starts with three 
students who appear to be hesitant in following procedures. The teacher interrupts and gets them 
started. She says, “Somebody read aloud so that everybody can listen.” 

One of the students reads from the screen what the teller reports about the situation. The 
teacher says, “O.K. Now, let’s read what it says at the top, at the yellow bar that you see at the 
top. And see what it says to do. Somebody read.” 
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One student reads the screen in a low voice and the teacher reads aloud, depicting the 
scenario situation. Then the teacher explains what the directions say to the students: The 
directions are to click on parts of the story that the students want to comment on and mark them 
either true or false. Starting at the beginning of the scenario, the teacher guides the students 
through parts: “So if you click on one of the lines....if you agree, you would click what?” 

One of the students answers, “True.” 

The teacher asks again, “If you didn’t agree, you would click what?” 

“False,” replies a student. 

The teacher goes on: “I think what you should do is to read what they are saying and pick 
true or false and tag it. Go through the lines and I want you to read each person — the customer 
and the employee.” 

As the students are reading a line, the teacher asks if the statement is true, and one student 
replies yes. Then, the teacher — ^without asking the students why they think it is true or if there is 
an alternative to that answer — tells them to tag true. The teacher directs, “Hit it again. Hit it in 
the middle. Bring it up.” 

They work their way through the scenario, a statement at a time, and then review the 
teller’s and then the customer’s perspectives — ^marking what the students feel is true or false. 

T his takes up all but about a minute of the segment. 

At the end of the segment, one student finally raises a question by saying, “The customer 
wants to cash a check for $ 4,000. That’s the problem. Is that a comment?” The teacher does not 
answer the student’s question, pressing for an answer for her own question: “Now, who do you 
think is right in here?” 

_ !l^tings for Segment 32 

♦Teacher involvement: 3 

General Categories Probiem-Solving Strategies 

♦ Use of Basic Skills: 1.5 ♦ Focused on the Problem: 3 

♦ Collaboration: 1 ♦ Recognize Confusion: 1.5 

♦ Problem Solving: 1 ♦ Make Predictions: 0 

♦ Use of Banking Concepts: 1 ♦ Previous Specific Learning: 1.5 

♦ Collaborate in Problem Analysis: 1 

♦ Previous General Learning: 0 

♦ Seek Help: 0.5 

♦ Make Changes: 0 

Analysis : 

Feeling the need to get through the details of the scenario and to train the students to use 
the true-false tagging tool, the teacher is imable to leave room for the students to collaborate or to 
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exercise problem-solving strategies. She is quick to direct the students in the activity. We do not 
know whether or not the teacher felt that these particular students needed extra help in the form 
of these rather explicit directions through each detail of the scenario; nor can we be certain 
exactly how the exercise ended, since the taping is cut short. 

At the beginning of the segment, the students’ facial expressions and relative silence 
suggest that they do not know what to do with the given task in the Mr. Smith scenario, which is 
scenario 10 and is relatively advanced. It is not clear whether they called for the teacher’s help or 
she saw that they were perplexed and came to help. Perhaps she was instructed before the taping 
to stay with the students all along. Whatever the reason is, the teacher in this segment seems 
focused on providing students with mechanical directions so that they can advance through the 
scenario lesson. 

The table above shows the average scores for the four categories and eight strategies 
across this segment. As can be seen, when teacher involvement is high, the scores for most of the 
twelve factors are low. It seems that the score for the category Focused on the Problem is high 
because of the high teacher involvement. Since the teacher gives directions on what to do, the 
students are not distracted, and proceed, using the true-false marking tool to assist their decision 
making. 
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SPigment 40 Scenario Number: 8 (Mr. Smith) 

Teacher involvement: 3 Length: 4.97 minutes 




Scenario Overview: 



This scenario concerns a conflict between a customer and a teller. The customer, who is 
dressed like a bum, wants to cash a check for $4,000. The problem arises when the teller says 
that she is not allowed to cash checks for this amount of money without approval from a 
customer service representative, and the customer complains about the bank service by saying 
that he is insulted. 

Description of Video Segment : 

The segment begins with the teacher asking a question. He asks the students if they know 
what to do with the screen. With negative responses from the students, he explains how to 
narrow down a choice: 

“What you can do is to highlight parts of it....If you highlight over there.. ..Just click on 
it.. ..You can highlight parts of the story. What you can do is, you can either put it as true or false, 
or you can make a comment about it.” 
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After giving a mechanical direction, the teacher probes for student involvement by asking 
many questions, such as, “What seems to be the problem? What’s your job? Why did they (the 
angry customer and the distraught teller) come to you (the service representative)?” and so on. 
Each time he asks questions, the students answer them: 

Teacher: O.K. Here you are presented with two sides. You hear two sides (the customer 
and the teller). What are you going to do? 

Student 1: We go to the manual and check it out on the good banking procedures. 

Student 2: We would know that she is telling the truth because she is not allowed to cash 
checks for more than 2,000 dollars. We know that part is true. 

Teacher: (Looking at Student 1) You brought up another issue. You said you have to 
calm the customer down. Why does the customer have to be calmed down? 

Student 1: (does not answer) 

Student 2: Because he is angry with the service that we gave him. He said that the way 
we treated him was insulting and stuff. 

Teacher: O.K. That’s the second issue that you are also going to have to deal with here. 
O.K.? Where do we go from here? 

Ratings for Segment 40 

^Teacher involvement: 3 

General Categories Problem-Solving Strategies 

Use of Basic Skills: 1 ^ Focused on Problem: 2.5 

^ Collaboration: 0 ^ Recognize Confusion: 1 

^ Problem Solving: 1 ^ Make Predictions: 0 

^ Use of Banking Concepts: 1.5 ^ Previous Specific Learning: 2 

^ Collaborate in Problem Analysis: 0 

♦ Previous General Learning: 0 

♦ Seek Help: 1 

^ Make Changes: 0 

Analysis ; 

While the teacher seems to ask useful questions to help the students get through this 
lesson, his assistance does not leave much room for them to discuss among themselves and 
collaborate. The ratings for collaboration and problem solving were very low, 0 and 1. For 
instance, even though the teacher’s last question (“Where do we go from here?”) seems to be a 
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useful one, it would have been more desirable if the students had come up with the same question 
by themselves. 

High teacher involvement also does not seem to allow leadership to emerge among 
students. For example, compared to segments involving further consideration of this same 
scenario and with these students but with lower teacher involvement, the students do collaborate 
some. The teacher is still present, it appears, and much that the students say is in response to him; 
but the teacher intercedes less often and the team does make entries as a unit and comes to a 
decision with some collaboration. The boy in the center, who is silent in the earlier segment on 
this lesson, begins to talk about the problem a bit. 

When student teams are left more to their own devices, they appear more readily to work 
together to solve a problem by asking questions of each other and by thus exchanging 
information. In one such segment, one boy exemplifies individuals in other groups by emerging 
as a leader, suggesting that when more dependent on themselves, someone in a student group 
assumes a leadership role. In the present segment, however, it is observed that none of the team 
members does this. The one boy sits back and says almost nothing, not even responding to the 
teacher’s questions. 

In short, when the teacher is highly involved, not all students are involved to a significant 
degree, there is little collaboration among students, and leadership is not developed. 
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S>f>gment 25 Scenario Number: 8 (Mr. Smith) 

Teacher involvement: 0 A Length: 8.35 minutes 




Scenario Overview : 

This scenario concerns a conflict between a customer and a teller. The customer, who is 
dressed like a bum, wants to cash a check for $4,000. The problem arises when the teller says 
that she is not allowed to cash checks for this amount of money without approval from a 
customer service representative, and the customer complains about the bank service by saying 
that he is insulted. 

Description of Video Sc2ment : 

This segment deals with a customer named John Smith who wants to cash a check.. 
However, Mr. Smith does not have any type of identification and looks like a bum. The three 
students refer to the manual to determine the procedures which apply to a customer, who does 
not possess an ID. The three students discuss which choice to make based on what the manual 
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says. They are confused because the manual does not direct them to a clear decision. Thus they 
try to make a decision by eliminating two choices and continue to discuss the other options: 

Student 2: "You don't have any proof that he's a bum." 

Student 3: "But he doesn't have an ID. I wouldn't cash the check." 

Student 1 : "Why wouldn't you cash it?" 

Student 3: "I wouldn't cash the check because first of all he doesn't have an ID and second 
no one has seen him before." 

The students are also shown discussing the consequences of their choices — some, as one 
member points out — after they have already decided to cash the check. In the conclusion of the 
scenario, they are delighted to learn that they have made the right decision — ^that the bum is a 
bank executive testing what would happen to a homeless person with no ID who had a legitimate 
check to cash. At the end of the segment, the students are a bit disappointed because they 
expected to get a promotion for not making a “bum” decision but the screen tells them that didn’t 
happen. 

Ratings for Segment 25 

^Teacher involvement: 0 

General Categories Problem-Solving Strategies 

^ Use of Basic Skills: 3 ^ Focused on Problem: 3 

^ Collaboration: 3 ^ Recognize Confusion: 0 

♦ Problem Solving: 3 ♦ Make Predictions: 1.5 

♦ Use of Banking Concepts: 3 ♦ Previous Specific Learning: 2.5 

^ Collaborate in Problem Analysis: 3 

♦ Previous General Learning: 1 

♦ Seek Help: 1 

♦ Make Changes: 0 

Analysis : 

Collaboration as a general category and as a problem-solving strategy is high throughout 
this segment with a rating of 5. It is observed as students are shown consistently interacting with 
one another. In addition, the students are shown taking turns in performing mechanical 
procedures (i.e., with the mouse and using the keyboard to write onto the screen). This segment 
is typical of what occurs when the teacher is not present for a sustained amoimt of time. During 
teacher absence, often students are highly focused, exchange their thoughts and perspectives in 
open collaboration, and demonstrate critical thinking skills. Furthermore, it is common to find a 
student taking a leadership role when the teacher is absent as one student does here. 




87 



Chelsea Bank Simulation: Problem Solving and Assessment Development 



Se gment 45 Scenario Number: 11 (Ms. Austin) 

Teacher involvement: 0 Length: 11.05 minutes 




Scenario Overview: 



This scenario concerns a young woman in her 20 ’s who would like the customer service 
representative to approve her car loan. 

Description of Video Segment : 

This segment begins with students discussing what they should do. It appears that the 
students are not sure if they should approve the car loan to Ms. Austen. Thus, they look at the 
rules in the manual. 

Student 1 : She has a little bit of money in the bank. . . . 

Student 2: Let’s check the manual to find out if there is any special rule we have to go by. 
One of the students clicks on the customer service icon. The other student reads a list of 
rules. However, they cannot find the rule they want. 
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They are shown discussing what should be their final decision. All three of them are 
working together and discussing actively each of their possibilities: 

Student 1 : O.K. She is a reliable person, which is true. She is responsible for financial 
matters. 

Student 2: What do you think? 

Cautiously, they consider the problem and how their decision could affect the bank: 

Student 2: O.K. Let’s go to the report. 

Student 1 : (Reads the screen) Define the problem. 

Student 2: (Writes) There is every reason to believe Ms. Austen is a reliable person.. . . 

At the end, they all agree that they should make the loan. It turns out, in the scenario 
discussion that this is a good decision. A highly successful person over the years, Ms. Austen 
never deserts the bank that put its trust in her. 
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Ratings for Segment 45 




^Teacher involvement: 0 






General Catenories 


Problem-Solvins Strateeies 




^ Use of Basic Skills: 3 


^ Focused on Problem: 3 




^ Collaboration: 3 


^ Recognize Confusion: 0 




^ Problem Solving: 3 


♦ Make Predictions: 1.5 




^ Use of Banking Concepts: 3 


^ Previous Specific Learning: 2 
^ Collaborate in Problem Analysis: 3 

♦ Previous General Learning: 0 

♦ Make Changes: 0 




r “ 

Analysis: 

i — 







The three students are extremely focused as they collaborate with one another in trying to 
solve the problem. Their previous specific learning is demonstrated significantly as the students 
refer to previous information that they have learned and utilized in the prior scenarios. Since this 
is one of the last scenarios students experience in the class, it is common to have some categories 
rated low. For instance, make predictions, recognize confusion, and seek help tend to be rated 
low as the students at this point have become familiarize with overall procedures and how the 
specific information they have picked up applies. 
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How clearly is an assessment component needed? 

As we studied the Chelsea Bank program, the need for an assessment component became 
increasingly clear. As we synthesized our analysis of 50 segments of student performance during 
the Chelsea presentations, our interviews with teachers and students in the program, and the 
analysis of the scenario lessons themselves, we determined that an effective assessment package 
could help both teachers and students recognize and focus on potential outcomes, increasing its 
ability to develop the skills, interpersonal behaviors, and other strategies that the study identified 
and defined. A good assessment package can also allow educators to hold the program 
accountable to its objectives while it serves as an impetus to school reform. 

It became obvious to us that for this to happen, both the students and teachers 
participating in it would need a clearer, firmer grasp on the goals and objectives they hope to 
achieve with the program. While the students, teachers, and other school personnel involved in 
the program were often enthusiastic about it and could cite some outcomes they expected jfrom it, 
many of them did not have a cohesive grasp of what its goals are or should be. 

Our observations of the major category Collaboration and its more specific application as 
a problem-solving strategy illuminate this point most effectively. We verified that students’ 
effectiveness in identifying and solving problems relied directly on their inclination to 
collaborate. When the members of a student team discussed among themselves the details of a 
scenario and the potential impact of a decision option, their behavior led more directly to an 
appropriate solution. Even when they were confused, collaboration helped student teams to 
recognize that fact and to decide where to go for information or other help. 

In the interviews conducted, students seemed to have a variety of conceptions as to what 
the program was achieving. Most commented on the real-world nature of the simulations, some 
detailing that they were learning some banking skills, the importance of following proscribed 
procedures and regulations, and how to get along with people. Some students recognized the 
importance of collaboration and noted that the program promoted it; but an equal number also 
bemoaned the need to consult with teammates about what they would do about the situations 
presented in the scenarios. Clearly, there was not an appropriate focus on collaboration among all 
of the students participating in the Chelsea experience. 

Nearly all of the teachers interviewed commented about the program’s potential to 
develop collaboration, but interestingly they were not often involved in promoting it. For 
example, when students decided that they were confused, they fi^equently turned to the teacher 
for help and jfrom that point on, student collaboration was replaced by exchanges between the 
teacher and one student at a time. So while some students recognized that learning to discuss 
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problems, issues, and options was a key lesson of the Chelsea Bank experience, that did not 
insure that they consistently demonstrated the strategy. 

Teachers were unsure at times as to how to guide their students when they needed 
assistance. Their monitoring usually consisted of helping the students continue through the 
scenario. Frequently teacher assistance directed students to the decision-making tools available 
within the program, but this assistance did not appear to be promoting team interaction as 
responses to teacher questioning and explanations. Turning back to the process of marking 
statements in the scenario as true or false, for example, sometimes ultimately led to resumed 
student collaboration; but most often, progress bogged down, awaiting the student interaction. 

The interviews with teachers suggest that they understand that developing student 
collaboration and teamwork is one of the program’s greatest potentials, but their recognition of 
that goal does not appear to be focused enough to direct instruction that will promote it. Rather, 
one of the major conclusions drawn from the observations was that student collaboration 
flourished in direct proportion to the absence of teacher input. 

Teachers who will be using the program will not have the potential to videotape their 
classroom as the program is administered and cannot be expected to analyze such data to 
recognize if it is achieving its goals. An effective assessment component, however, can enable 
them to determine if the Chelsea Bank program is indeed developing collaborative behavior 
among their students. It will direct a focus on this and the other goals and objectives that the 
study has identified for the program, and this will effect both instruction and student responses to 
it, while revealing its effectiveness in their classrooms. 

Our study revealed that while some teachers were aware of this and other goals of the 
program, most were not sure of its multiple potentials. Our analyses of the taped segments of the 
program in use reinforced our identification of all four major categories as program goals and of 
the eight strategies that appear reasonable and important in identifying and solving the problems 
arising from the banking scenarios. Some of them were more evident in the student behaviors 
than others. The need for some was underlined both by their presence in student performance and 
by their absence. Some were rarely seen but of apparent potential. Assessments can be developed 
to ensure that teachers and students focus on these potential outcomes. 

Several teachers who were interviewed noted that assessment is an important component 
that is missing from the program. They knew that their students were engaged and interested and 
that they were completing the simulation lessons, but they seemed unsure of just what the 
students were learning. A few developed their own assessments — usually tests covering banking 
concepts. They often expressed their need for the information that it would provide them in terms 
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of particular content and behavioral goals that are discussed in the following section of this 
report as factors that relate to designing an assessment component for the Chelsea Bank 
program. 

After e xaminin g the entire program, carefully observing students and teachers using it, 
and interviewing teachers and students, we continue to be impressed by the intensity with which 
the students engaged in the simulations. However, if this program and/or the approaches on 
which it is based are to be adopted, decision makers will not be able to invest the time and focus 
to examine the program as we have done. They will want assessment data that reflect what their 
students are getting from it as well as information that will assist teachers and students in getting 
the most out of it. 

What kind of assessment may be designed? 

While noting the need for an assessment component during our study of the Chelsea 
Bank program, we were able to identify the purposes that it should serve; and we were able to 
consider the kinds of assessments that would serve those purposes and what the content and 
structure of the assessments might be. 

It has been determined that assessment must first serve to focus the simulation materials 
on expected outcomes so that teachers and students alike can have a clear grasp of the potential 
benefits of using the program. Assessment will need to reveal how well students perform, 
indicating how readily and how effectively the students collaborate, how well they apply various 
strategies to solving problems and making decisions, how well their command of basic skills 
serve that application, and to what degree they have gained a command of banking concepts. 

It seems clear that the Chelsea Bank assessment will need to report on the processes the 
program involves and nurtures, not just to test the retention of terms and information. Our 
analyses revealed that the assessments developed should have multiple purposes, serving 
multiple audiences. This will require measures and approaches that can be used at different times 
in the program. 

Three major and over-riding emphases of these assessments will need to answer these 

qt » c • 

• Do the learners grasp basic information and concepts presented during the Chelsea 
experience? (The assessment will need to measure knowledge gained.) 

• Are the learners able to apply what they have learned to solve other problems? (The 
assessment will need to examine how well the program has met its goals in application.) 

• Are the learners able and inclined to assess their own skills, behavior, and progress? (The 
assessment component will need to include aspects that will model and develop the 
student’s tendency and ability to self-assess.) 
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The assessment component must meet the needs of three major audiences: 

• It must serve the students, focusing the goals of the program as expectations of its 
potential. It must provide indicators to the students both of what they are learning and 
how they are developing and of how they can leam more — become more effective 
thinkers and problem solvers, know more about banking, and develop and apply their 
basic skills to similar tasks. At the same time, assessment must involve the students in 
monitoring and analyzing their own progress while modeling approaches and methods for 
doing that. 

• It must serve teachers . It should not only indicate to teachers whether the students are 
learning, but also how they are learning. Teachers want and need to understand the 
processes that students are using effectively and which need development and practice. 
The assessment component should indicate directions and emphases for other instruction 
and classroom experiences. 

• It must inform decision makers . The Chelsea Bank assessment needs to account for how 
well the program is achieving its goals with the particular groups of students using it. 
Assessment should reveal the program’s impact or lack of it. 

Assessing for the Chelsea Bank program should take place at various points during the use 
of the lessons: 

• Needs assessment should serve the preparation and planning of the Chelsea instruction, 
both related to and growing out of the Chelsea scenarios, and this same aspect of the 
assessment component can serve as a baseline for assessing growth. 

• Assessment needs to enable the monitoring of student progress, in effective strategy use, 
for example, as the series of Chelsea lessons progresses. 

• A summative facet of the assessment package must report on the accomplishment of 
program goals when the lessons are completed. 

These three general emphases create a kind of dimensional approach to designing the 
assessment component by allowing the research team to consider how these three considerations 
interrelate and interact. For example, in a structured model of the three-dimensional approach, 
the cell that places the focus on the self-assessment of students to monitor their own performance 
suggests the possibility of informal checklists with questions about both what they know and 
how they are performing. Such tools in an assessment package might well be overlooked without 
this type of a planning approach. This is but one example of how the matrix will promote 
considerations about the types, content, and structure of elements of the assessment package. 

Obviously, the designing of the assessments will rely heavily on the awareness of the 
program developed during the first-year study. What we learned from the analysis of the lessons, 
the observation and analysis of the students’ performance on the videotapes, and the teacher and 
student reactions and suggestions gained from the interviews will help guide these complex 
considerations. 
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The eight strategies that were articulated for studying the main category of problem 
solving will figure in the assessment of what students learn from the Chelsea experience. The 
performance perspective will surely involve some report on how the student’s command of basic 
skills affects his or her problem-solving ability. It is yet to be decided how the students’ grasp of 
ha nkin g concepts will be assessed — ^with a kind of test, as some submeasure within a 
performance assessment, or in some design that combines the two. 

But while the twelve considerations that structured the analysis of the tapes guided the 
team’s grasp of the program, they may not form or dictate the primary structure of the analysis. 
The consideration of how the program can best be assessed began and continued during the 
different analyses and are a major activity bridging the efforts of year 1 and year 2 of the overall 
study. 

The interviews demonstrated the complexity of the considerations, and as they struggled 
for a firmer grasp of the goals of a program they foimd so involving, both teachers and students 
contributed to rich vein of possibilities. One teacher recommended a traditional form of testing to 
find out what the students are learning; another suggested that assessment might grow out of her 
approach of listening to student conversations to find out when they are having problems. Other 
teachers recommended content emphases, including math and banking concepts. 

A student, who thought the student groups should get a “group grade,” noted that there 
were no tests but that “Every day was a test for me — ^we had to do new things.” Another student 
recommended both individual and group tests. 

The range of one teacher’s suggestions suggests the need to be open in considering what 
assessment can be developed: 

“If I were to design a test for [the] simulations, it would [cover]... math skills; [banking 
concepts like] what is a ‘good check’ and procedures for doing things in a bank; [and] 
personal matters. I [would want to know] if I gave them a problem (not [necessarily] 
workplace), could they solve it? What kind of problem [could they solve]? I would test 
them individually to see what each student got out of it. No group assessment. I think 
they should be tested on ethical issues. They need to follow procedures and [to] run an 
ethical business. . ..I would. . .develop team portfolios.” 

Some of this teacher’s input is reinforced and enlarged by that of another: 

“If I [were] to develop a test for the simulation, it would present new bank problems to 
determine how students might solve [them]. I think [we] would need both individual and 
group tests to see if students [have] learned to work together and to see if they [have] 
individually learned new concepts. The assessment should definitely be administered on a 
computer.” 
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These excerpts of interview data indicate the scope of the considerations that will initially 
frame efforts to design the assessments. Our understanding of the program has grown with 
ongoing analysis of the lessons, scenarios, and the response of students and teachers to them. The 
results of this effort will be directive as the assessments are designed, tried out, and perfected 
during the second and third years of the project. 

The experience has led to ongoing brainstorming among members of the research team 
with a host of relevant but unrestrained ideas that exist in notes and our minds as considerations 
to be discussed and considered. Just a few of the ideas that they include are: 

• Experiment with having the participants do some kind of “process writing” as they solve 
problems so that it can be analyzed using some “large-scale” assessment of the strategies 
the student is aware of and using, the problems that the student has encountered, the 
number of banking concepts that are understood and used correctly, etc. 

• Have the student record reactions to the lessons and respond to questiormaires or other 
checklists and self-analysis ratings that make up a kind of job-preparation portfolio. 

• It seems reasonable that the assessment environment should allow and even require 
collaboration and be administered in a setting that encourages it. We must consider the 
possibility that all — or a significant part — of the assessment will need to be administered 
to teams, as opposed to, or in addition to, individuals. 

• A part of the assessment package can assist, guide, and promote meaningful teacher 
observation that can be correlated to analytical suggestions about what different things 
seen may mean in terms of student development. 

• W hil e a test of (or assessment involving) banking concepts would need to rely on those 
that the program has presented, it might introduce one or more new ones that can be 
understood in an inferential way if one has a grasp of some others that were presented in 
the scenarios. 

• Like a few of the scenarios, an assessment problem/scenario might not be resolved by 
finding a particular guideline in some tool or even by combining two or more bits of 
available information. It might involve considerations that force collaboration as to a 
judgment call. If this is tried, it seems reasonable that there not be one “correct” response 
or solution; this part of the assessment would focus on the process/behavior of the 
student. 

• The solutions of the problems need not always be presented as multiple-choice options as 
in the scenario lessons. This might be a more difficult, but better, test of decision-making, 
problem-solving behavior the student has developed. 

• One or two scenarios might be targeted as “open-ended” in which the student writes one 
decision and/or several possible outcomes. 

• The team should be fed some results or information in a step preliminary to the final 
resolution, so it has an opportunity to change its decision or to add an option to possible 
solutions. Perhaps this could be done in a scenario as an intermediate supervisor’s 
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recommendation. This would turn an aspect of the scenarios that the students seemed to 
enjoy a great deal, but it would make it instrumental to a test of particular strategies — 
making changes/seeking help — ^that they might use. 

• There should also be some intermediary feedback that follows some incentive/motivation 
that requires the student to make a prediction, which could be followed by an opportunity 
to make changes based on that feedback. 

• A similar intermediary step should allow students a chance to articulate/recog/j/ze 
conjusion and to let them seek help by specifying a particular information source. This 
would help teachers target needed instructional intervention. 

• Many of these ideas suggest that a major portion of the assessment should be presented in 
steps/stages/sequenced events interlaced with feedback or the presentation of additional 
information about the scenario not available until then. 

• Using the tools must be considered an important type of seeking help. Otherwise that 
strategy is going to over-encourage reliance on the teacher. The teacher’s role must be 
specified in some aspects of the assessment. 

• To test keeping a focus on the problem, some attractively distracting element can be 
incorporated in one or more scenarios. 

• The students could be required to keep portfolios in which they keep notes about 
problems, reactions, etc. An ongoing account of the experience could be included. 
Students could be provided materials to provoke and guide several perspectives in self- 
assessment. It could include some scenario extensions coded to responses and handed out 
by the teacher after decisions are made — similar to the promotions, compliments, 
criticisms they got with the scenarios. This would be some solution to the need for 
individual assessment within a component that administered some aspects to teams. 

• The students could be required to keep journals of their experience with the program. 
Some very general guidelines could suggest comments about the student’s relationship 
with other team members, matters that seem confusing or unresolved, etc. These journals 
might be kept in a portfolio common to the team. 

• The members of a team could keep a response journal in which they get a chance to 
review and elaborate on interaction in collaboration. This might also incorporate a way to 
record any increasing reliance on the strategies recognize confusion, seek help, make 
changes. 

• All of these possibilities involving open writing, portfolios, and journals would be most 
effective if the teacher were provided with some suggestions and help in interpreting 
entries. 

• If a part of the assessment were administered on computers, the students could be allowed 
to ask questions to be answered by a file like that usually called “Help” in most software. 
It could have a search capacity that picked up on key words in questions typed in. 

• A test of banking terms might involve some options that attempt to appear like 
application. 
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• A test could also give the student a list of information sources/tools and then several 
problems/questions, requiring the student to indicate where s/he would go for an answer. 

• Some assessment items or scenarios should place the student in the position of supervisor 
and require him/her to react to the performance of some teller or junior loan officer. 

These are merely a few jot-note type ideas that indicate the rich pot of considerations to 
be reviewed early in the process as the development of the assessment begins. It is probable that 
a broader array of assessment tools may be in the development stage early in the process before 
the final mix and assessment component package begins to take shape to be fully designed for 
the tryout stage. 
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V. Teacher-Student Interactions During the Simulations 

Simulations like Chelsea Bank significantly impact the role of the teacher in the learning 
environment. Simulations place the learner in the role of decision maker and problem solver. 

The content to be learned is encountered in the process of problem solving. Thus the role of the 
teacher can no longer be that of presenter of information. Instead the teacher must work with the 
students as a coach or guide, supporting their problem-solving skills. But it is not clear how 
teachers should coach. The Teachers Guide accompanying Chelsea Bank does not give the 
teachers any guidance as to what they should do while the students are working on the 
simulation. Further, as we noted earlier in this report, there is not much experience in using 
simulations in the school environment and therefore there are not standards of practice available 
in the literature. 

Given this context, we sought to understand how teachers using the program interacted 
with their students. We identified three models of coaching. First, there is the "stay out of the 
way" coach. In this coaching style the teacher focuses on getting the students started with the 
problem, motivating them, and then getting out of the way. There is very little student-teacher 
interaction once work begins. The second model of coaching is the teacher-centered model. In 
this model, the coach tells the students what to do. This coach will remind the students of the 
"correct procedure" and tell them how they should proceed. Such a coach will also observe the 
learners to make sure they are doing it “right.” The final coaching model is the learner-centered 
model in which the coach asks the students questions and challenges their thinking. The support 
is based on, and arises from, the student's thinking. 

These three models can be distinguished by two criteria: the stimulus for interacting with 
the students and the pattern of information exchange. The coach who stays out of the way will 
only interact with the students when called upon, while the other coaching styles will include 
unsolicited interactions. However, the learning- and teacher-centered coaches can be 
distinguished by the directiveness of the teachers’ comments to the students. 

The goal of the research reported here was to examine the student- teacher interactions with 
regard to the frequency, stimulus for, length, and nature of the interaction. From these data, we 
seek to infer the coaching strategy of the teachers. 

Method: How was the data for this analysis collected? 

In this section of the report we turn to the new classroom data, i.e., the classrooms where 
we worked with the teachers and Classroom Inc. to collect video records. Four high schools 
(EBC, Salesian, Jefferson, and King) participated in this data-collection effort. Each high school 
submitted tapes of students working on scenarios 3, 8, and 1 1. A total of six classes were 
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represented on the 15 video tapes which provide approximately eight hours of footage combined. 
Each student-teacher interaction was logged and evaluated. 

Several if not all of the teachers are engaged with the student groups before and after the 
scenarios. While this is relevant to the analysis of coaching strategies, our video record is limited 
to the actual time the students were working on the scenario, and therefore our analysis is limited 
to the actual time students are working on the scenario. 

Findings: What did we learn about student-teacher interaction? 

The results of the study depict teacher-student interaction with quantitative analysis and 
qualitative description: 

The quantitative analysis does not reveal high frequency. 

Table 5.1 presents the frequency of teacher-student interaction during the scenario at each 
of the schools. Across the 18 Chelsea Bank scenarios there were only 34 instances of the teacher 
talking to the group. The student-initiated interactions tended to be cases where the students 
would ask a question such as, "Can you make a check out to cash?" or "How do you spell 
Rodriguez?" In most cases, the teacher answered the student and moved on to another group. The 
teacher-initiated discussions were generally orders being given by the teachers about the way a 
scenario should be solved. The “other” category included both disciplinary (e.g., the teacher 
telling the students to get back to work) and interactions that had nothing to do with the scenario 
or the students work on the scenario (e.g., discussion of other projects.) 



Table 5.1 Types of Teacher Interactions with Students 




Instructional 


Other 




Schools 


Student- 

initiated 


Teacher- 

initiated 




Totals 


Jefferson (2) 


7 


2 


1 


10 


EEC (2) 


1 


0 


2 


3 


King 


6 


2 


1 


9 


Salesian 


8 


3 


1 


12 


Totals 


22 


7 


5 


34 
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Twenty-nine of the thirty-four interactions were instructionally based. Thus, the teacher 
only approached a group for instructional purposes 1.6 times per scenario. The median' length of 
the interactions is only 23 seconds. Therefore, the teacher interacted instructionally with the 
students an average of 38 seconds per scenario, whereas the scenarios generally took 35 to 40 
minutes to complete. 

Of the twenty-nine instructionally motivated student-teacher interactions, only seven were 
initiated by the teacher, an average of one visit per 2.6 scenarios. There was variation across 
teachers: The teacher at Salesian initiated an instructional interaction once per scenario, while the 
other teachers initiated an instructional discussion once or never during the three scenarios. 

These suggest very strongly that the dominant model of guide is to “stay out of the way.” 
There is little, none in one case, instructional monitoring by the teacher. 

We next looked at how the teacher responded to the student’s request for information. In 
most of these twenty-two instances the teacher was asked a question that the students could have 
answered on their own with some guidance. However, in most cases (77%) the teacher provided 
an efficient response and moved on. Only five cases out of the twenty-two student- initiated 
interactions (23% of the cases) involved the teacher providing an opportunity for the students to 
grow and learn from their question. The "Other" category in Table 2 reflects instances where the 
teacher either did not respond to the student’s question or responded by asking a question such 
as, "What did you pick?" and walking away. 



Table 5.2 Teacher Behavior in Student-Initiated Interactions 




Directed/ 

provided information 


Guided 


Other 


Jefferson 


5 


2 


0 


EBC 


0 


1 


0 


King 


2 


1 


3 


Salesian 


6 


1 


1 


Total 


13 


5 


4 



1 We report the median here because the scores are skewed in part due to a basement effect but also due to one 
extreme outlier ~ a 10 minute interaction with the students. The mean length of the interaction was 57 sec. 
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In summary, the teachers tend to stay out of the way while the students work on a scenario. 
However, when they do interact with the students, they tend to be guided by the teacher-centered 
model. When teachers are initiating an interaction, it is usually to give instructions to the 
students. When the interaction is student-initiated, the teacher typically responds by simply 
providing the requested interaction. When they don't just provide information, teachers are 
almost as likely to ignore the question (the “other” category) as they are to engage the students in 
a learning dialogue. 

A qualitative description depicts the nature of the interaction. 

The description in the previous section provides data that suggests that teachers tended to 
be directive in their interactions with students. In this section, we attempt to provide a richer 
flavor of the nature of that interaction. In the following pages there are summaries of the 
interactions in each of the four schools sequenced from the site that was most learner-centered to 
the site that was most teacher-centered in terms of the pattern of interaction. 

Each section begins with a discussion of the teacher's style with examples of that style. We 
then present a sample of interactions chosen because they represented typical interactions for that 
teacher. The scenario descriptions are preceded by a number that refers to the Chelsea scenario. 

In cases where more than one class was used at a school, the class number is also provided before 
the description. (For instance, "3.2" means Chelsea scenario number 3, second class for the 
teacher.) 

Kinff Hieh School . The teacher at King High School showed the most evolution during the 
course of the three scenarios. In the early scenarios, she used the step-aside approach almost 
exclusively. By scenario 1 1, however, she had evolved to a learner-centered approach. She 
stimulated the students' thinking with hard questions and encouraged them to discuss their ideas 
even when she knew that they were "wrong" according to the answers provided in the scenario. 

In the most impressive discussion from any of the video records, the teacher at King tries to 
help the students express their thoughts and understand where they made their errors. It occurs 
after the students have called the teacher over because she told them to let her know before they 
answered the discussion questions in the simulation. (In the transcript, T= teacher and S = any of 

the group members) 

T: What was your reason for picking C? 

S: She's young; she wants a sports car even though she wrecked two in high school. 

T; What do you think, Antoine? 

S: Yes. 

T: Why? 

S: Like my partner said, she's young, when in high schools, she hit two cars. 

T: Do you want to go with C also? 

S: Yep. 
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T: Why? 

S: She's about 22. She asked for over $10,000, and she may not even be over 20. 

T: There were two right answers. 

S: C&D— watch! 

T: One was A. 

S: See I told you. .. . 

T: You said you agreed with C. 

S: I said A before we even did this. 

T: What do you think the other one [correct answer] was? 

S: D. 

T: Why do you think that it was D? 

S: Did we pick the right answer? Yes or no? 

T: Why do you think it was D? Do you know what D was? (Reads part of D to students) 
S: She's young — she still seems wild. 

T: So, do you think it’s D? 

S: No, I still say its C. 

T: Antoine? 

S; What's the two right answers? 

T: AandB. 

S: What? 

T: (Reads A, then reads B) Do you think Ms. Austin's age should have been a factor into 
the decision about this loan? 

S: In a way. 

T : Why should it be a problem? 

S: We don't know her age. She doesn't have much experience. 

T: Did you check out her finances? 

S: Yes. 

T: Did she have money in the bank? 

S: She has $880 now. 

T: What do you think, Gerome? Should her age be considered? 

S: Yep. 

T: Why? 

S: The girl looked young. 

T : So why should that enter into whether you give her a loan or not? 

S: It interferes — she can pay it, but how do we know if we can trust her or not? 

T: Well, if she was 40 years old and had the same account, it would have been all 
right?" 

S: The point is that she had two cars and she trashed them. 

T: Do you think its appropriate to ask if she ever owned a car before? 

S: Yes. 

T: Why? 

S: If you get money, they should ask that. 

T: Why? 
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S: They're giving you money. Let's say they have a Honda and they want to go get a 
Mercedes just cause they want a Mercedes. They should ask. 

T; What do you think, Gerome? 

S; Same thing might happen — ^they might crash. They gotta know how you took care of 
the car. 

T: Is it appropriate to ask if the person had a loan to buy a car before? 

S: Yeah, cause that's when you're trying to figure out if they're trying to bum you. 

T: Should the information about the two cars Ms. Austin had in high school being 
wrecked influence the loan? It did in your case, didn't it? 

S: Umhmmm.... 

T: Do you think Ms. Austin handles money wisely? 

S: She's single and has an apartment and buys clothes and stuff. 

T: We're going to have to shut this down for now — you can finish tomorrow. 

Even though the students show no sign of reaching a better imderstanding of the flaw in 
their decision, the teacher remained very patient and tried to lead them to imderstand what 
happened. Unfortunately, when the students finished the scenario the next day, there was no 
further discussion of this. It is this kind of conversation, which lasted less than eight minutes, 
that will help the students grow intellectually. 



Table 5.3 Selected Interactions at King 

Incident : 

(3) A student says, "It says 'Pay to the Order of Cash.' Can you do that?" The teacher does something to 
indicate that you can. 

Interpretation : 

The students express a reasonable concern in evaluating the banking transaction. However there is 
no information in the system that can provide the answer and it is not a matter of logical inference. 
Therefore, we see this as an instance where it is reasonable for the teacher to provide a direct 
answer to the question. 

Incident : 

(1 1) One student comments, "They don't say how old she is." The teacher responds by asking if they 
looked at her ID. One of the other students in the group says, "I say she's about 22". 

Interpretation : 

The teacher offered an option for finding out how old the person is. This seems appropriate — she 
doesn't ask why they want to know or tell them that they don't need to know this. She tries to let 
them figure it out for themselves. She chooses not to persist when the students ignore her 
suggestion. 
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EBCHieh School . The teacher at EBC displays elements of all three models for coaching. 
The main examples of an inquiry- or learner-centered strategy can be found in the lead-in and 
follow-up discussions with the whole class. Following is an example of the nature of the 
conversation at the end of the third scenario. The students speaking are the group we watched in 
the morning class. 

When the group finishes, they cover up their monitor with a sheet of paper because they 
are embarrassed about getting the wrong answer. The teacher asks (as part of whole class 
discussion) “What happened?” 

The students say, “We got fired” and “We had to go to court.” 

The teacher continues, asking, ’’Where did you go wrong?” The students say that they 
should have had the customer sign the check in front of them. One group member 
mentions that this never happens to him in real life — ^they just look at his ATM card. The 
teacher asks what the student thinks they want the ATM card for. 

He says, “to check the account.” The teacher points out that it has a signature and is a 
form of ID. Then she asks class to explain what might have gone wrong with this 
scenario. 

This is a common kind of conversation in the large class discussions following each 
activity. The teacher obviously wants all the students to think about what they are doing and 
provides a discussion that pushes the students. 

This teacher, however, is almost absent from the video when the groups are working on 
Chelsea. Occasionally, we see her walking around and glancing at the work the students are 
doing. Unfortunately, in some circumstances, the teacher’s unwillingness to coach or help the 
students can actually be an inhibitor. One instance in particular stands out: 

The students call the teacher over and say, "None of the answers make sense" and then 
explain what they mean. 

The teacher says, "What do you think you should do?" The students say that the check is 
payable to “Cash” and they think that's wrong. They explain that they think a check 
cannot be made out to cash because a check needs to be written out to a person. 

Then, the teacher asks, "What's cash? What's the definition of cash?" The students try to 
look it up in the glossary, but it isn't there. 

One of the students offers, "Cash is money." 

The teacher responds, "What's money? Think about it. . .." and walks away. 

In this case, the students were stuck on an issue that should not have become a major 
concern of solving the problem. The teacher was trying to promote inquiry and help guide the 
students to new knowledge, but instead left them fimstrated and confused. This seems to be a 
potential problem for any teacher learning how to facilitate problem-based learning situations. 
They must learn to discern between opportunities to foster growth and getting in the way of 
learning. 
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Table 5.4 Selected Interactions at EBC High School 

Incident : 

(3.2) The students say they are done and tell the teacher that “That's 2 satisfied customers in a row.” 
The teacher asks what happened with the first one. The students say they got it right too, but she 
missed her brother's wedding, so the customer wasn't happy. 

Interpretation 

The teacher expects the students to remember what they've done — ^that seems good. The teacher is 
not expecting much beyond just memorization of events. 

Incident : 

(11.1) This tape starts out with a long whole-class discussion introducing the ideas for getting loans. 
The teacher asks things as if the students were going to do it in real life. For example: “What if you 
were going to get a loan?” A little while into the conversation, the teacher notices that people in one 
part of the room are not paying attention; she starts specifically calling on them instead of other class 
members. The group we are watching quickly loses interest in the conversation and starts talking about 
other things. 

Interpretation : 

She is starting by preteaching for the scenario that completes this activity. Because the teacher is 
focusing on getting a certain part of the class to participate, she excludes some students, such as 
the ones in the group we see. They quickly lose interest. (It also appears that they already know 
the information being covered which may also have lowered their engagement level.) The 
students become engaged once they begin work on the scenario. 

Jefferson Hish School . Jefferson High School provided an unusual circumstance. The 
regular classroom teacher is not involved in the simulation. The person who acts as the teacher 
does a good job of trying to ask good questions and aid the students in learning. However, he 
never gets into the questions that would really make the students think, such as "Why?" He tends 
to ask the students to summarize what they have done. For instance, in the third scenario, the first 
group finished and the teacher asked what the problem was in this scenario. 

One student explains that someone brought in a check form a different branch, and they had 
to decide whether or not to cash it. The teacher then offers, "you just needed a signature 
card?" The students tell him that they needed two IDs. Then the teacher asks if the 
customer was a friend. 

The students say "No, but the customer was well-dressed." The teacher laughs and thanks 
them. This is a fairly typical follow-up dialogue at this school. While it confirms that the 
students did complete the scenario, it does not push them to reflect on what they decided. 
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The teacher’s questions also suggest that he is not very familiar with the scenarios. 

However, he does not seek to learn from the students but simply drops the query. 

The strictly hands-off approach used in this situation does allow the students to develop 
their own ideas about how to solve problems. However, the students never get any guidance to 
help them understand where their plans need to be enhanced. 

Table 5.5 Selected Interactions at Jefferson High School 

Incident . 

(3.2) The students are discussing their inability to get information on the customer. The teacher 
explains that the credit check is for loans and this customer has an account so they need to check 
"accounts." The students type the name again, and the teacher, again, says they need the accounts page. 
The students say they tried that, then mention that they caimot read the customer signature well enough 
to spell the name. The teacher has them back out of the screen and shows them that the customer's 
name appears in print on the check. They go to accounts. The teacher points out that savings accounts 
are on one side and checking is on the other. Then he asks if the students are sure of the procedures and 
suggests that they "might want to check them." 

Interpretation : 

This is a case of teacher as information. He was watching and realized that the students were stuck 
(they looked frustrated and weren't sure what to try next). Rather than asking questions or 
modeling problem solving, he simply offered suggestions as to what they should do. While he was 
trying to be helpful, he gave them solutions rather than the solution-finding process. There is 
another question here: “Why is it that in the third scenario the students still don't know that on the 
screen one side is checking and the other is savings?” 

Incident : 

(3.2) The students ask how they can ask the customer for identification. The teacher says they'll have to 
go on to the decision area in order to do that because that's where that is pertinent. He also mentions to 
read all the decision options before picking one. 

Interpretation : 

The teacher actually helps the students work outside of the authentic problem — suggesting they 
look to the solutions to see what information is needed. Yikes! 

Incident : 

(8.1) The teacher discusses why the students made the right choice: All they needed was a signature 
card because the bank account was at Chelsea. He says "Good Job" and asks if they need anything 
else. Then says, "Thanks." 

Interpretation : 

This could have been an opportunity for the students to express why they chose the correct answer, 
but the teacher instead stepped aside and let the students go. 
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Incident : 

(8.2) The students are puzzled because they think C is the right answer except that the last sentence is 
rude (telling the customer that he needs a bath). The teacher says that they need to make a judgment 
because they have "to accept the whole answer, not just part of it." 

The student says that its about the ED that the guy doesn't have, but they can't tell him he stinks. Again, 
the teacher says that they will need to use their judgment: "You can't choose a part." The students ask 
once more. The teacher says, "Choose the one that is most correct." 

Interpretation : 

This illustrates how providing decision options in Chelsea Bank disrupted — invalidated — ^the 
students’ problem-solving process. To deal with this, the teacher would have had to step outside of 
the Chelsea Bank framework and let the children formulate their own answer — and then have them 
compare it to what Chelsea said. However, he clearly did not feel comfortable assuming that role 
and chose instead to make it an academic decision (choose the whole answer) rather than 
promoting a real problem-solving decision. 

Incident : 

(8.2) Teacher says, "Remember, I told you to read the whole thing." 

The students discuss their choice. One says, "See, I told you." 

The teacher says to read the other consequences. The students are puzzled by the correct answer. One 
says that th inkin g you're going to faint is just as bad as saying to take a bath. The teacher points out that 
"you think that, not say it." Then he turns to the problem and asks if the problem was with the check. 
The students say it was with the ID. The teacher asks what bank the check was form. They say it was 
written to the guy. 

The teacher says, "Did you check the account?" and "Did it have money?" 

(Students say "yeah.") 

The teacher says the check was from Chelsea so they didn't need ID. 

The students respond, "So we get fired, or quit — that’s not fair because the first [correct answer] was 
rude too." 

T: But you thought that. Do you see the difference? 

S: He didn't have any [ID]. 

T: Did you check his signature card? 

S: Yeah. 

T: It was his signature and he signed it in front of you? 

S: Try to say what rule book said. 

Then the teacher tries an example from "real life" explaining what would happen if you go into a bank 
like this. One of the students says, "Oh yeah, my mom does that and she doesn't have ID." 

Interpretation : 

The teacher is definitely trying to promote inquiry here. He is trying to explain, without dictating 
or directing, why the answer is incorrect. He uses a few different approaches to help the students 
see why their answer was incorrect. 
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Salesian Hish School . The teacher at Salesian was easily the most director-like of any of 
our teachers. He consistently acted as an information provider. In fact, almost every instance of 
interaction with the students involved him telling them what to do next. This teacher is also the 
only one who specifically assigns tasks to the students. Each student has a particular "job" within 
the group — one tj^es on the simulation computer, one tj^es the rules on another computer, and 
the third is best termed the “information manager.” 

In one tj^ical instance, the teacher has a step-by-step procedure for his students to follow 
and refers to it occasionally when they ask him questions. For instance, in scenario 3, the 
students ask him if the page of the manual they are looking at is the correct page. 

Mr. Zacc responds by saying, "It says page 8 and 9. Go to page 9." He continues, pointing 
out that "Step 2 is to obtain proper ID." Then he says they need to check page 8 if they 
don't know what that means. Then the students ask if the steps listed are the steps for 
cashing the check. Mr. Zacc says, "Yes" and proceeds to explain each of the rules to the 
students. When he walks away, the kids start typing the rules, word for word, into the log 
that they are required to keep. In this situation, Mr. Zacc first told them exactly where to 
look for information, thereby preventing them fi"om developing their own process. Then he 
immediately started explaining what was written in the manual. This furthers the 
relationship that is clearly set-up in the classroom where the teacher is the holder of 
knowledge and the students are dependent on him to dispense that knowledge. 

In an example fi"om the 1 1th scenario, Mr. Zacc shows no sign of progress fi"om the 
mindset in scenario 3. The students call him over because they do not know how to fill out part 
of the loan application. 

Mr. Zacc immediately tells them they cannot do that part of the loan application fi"om the 
screen they are on. When the students ask why, he points out that the information they need 
is fi"om the credit check screen and they are in the account information. Then he says, "This 
information has to come fi"om the computer. Remember, I showed you the steps? When you 
get to that step, you fill out this information." This final statement confirms that he has not 
allowed the students to create their own problem-solving model. Furthermore, when the 
students asked "Why?" Mr. Zacc was given an opportunity to help them think through the 
situation to arrive at the answer. Instead he just gave them the answer. 

Mr. Zacc's class is interesting in some good ways as well. Because the video often ran over 
into the part of class immediately following work on the scenarios, we were able to hear the large 
group follow-up discussion and the assignments he gave based on the Chelsea Bank scenarios. 
The assignments were writing assignments based aroimd reflection on the scenarios. For 
instance, the assignment after scenario 8 made the students consider the following questions: 

• Have you ever been treated rudely? How did it make you feel? 
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• Do you think there are circumstances where it should be allowed to turn away Mr. 
Smith? Where? When? 

• When working at a job, do you have the right to act on your own opinions of customers? 
Why or why not? 

• Do you think your attitude toward and treatment of customers can effect your business? 
These questions all provide some opportunity for the students to reflect upon the Chelsea Bank 
activity. 



Table 5.6 Selected Interactions at Salesian 



Incident . 

(3) The teacher walks by and notices the students typing account information. He says they have to 
figure out what the transaction is, then asks them what it is. They reply, "Cashing a check." Mr. Zacc 
then tells them to read the manual first, then look at the account information because they can't know 
what's im portant without checking the manual. 

Interpretation : 

T his is definitely a case of teacher as information provider. Mr. Zacc has laid out a specific order 
in which the activity should take place. He discounts that there can be any way for the students to 
have any ideas without consulting the manual. 

Incident : 

(3) A student asks, “Mr. Zacc, how do you spell her name?” He spells it as he walks by. He gets 
almost past and backs up. Says, “You’re in credit check” and points out that there are two things to 
check. He says they need to look at accoimts. Student mentions that they’ve checked three times. As he 
walks away, they ask him how to spell her name again. He looks at the machine they are taking notes 
on and points out that they have it. 

Interpretation : 

Not surprisingly, in cases where the teacher provides all the answers, the students start becoming 
quite dependent on the teacher. By telling them they were in the wrong section, Mr. Zacc saved 
them a lot of time, but there is no clear evidence that they really imderstood why they were in the 
wrong place. 
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Incident : 

(3) Student asks, "Does she have to sign again?" Mr. Zacc says that its contusing because they don’t 
know if she signed it there or elsewhere. They say her signature matched. He says good, then asks if 
the quietest group member agrees. (Everyone in group says yes.) 

Interpretation : 

This is a good question and an appropriate answer. Mr. Zacc tries to explain the shortcoming of 
the progr am to the Students rather than merely giving them a yes or no answer. This seems to help 
build more understanding than just saying "yes" or "no." This is a case where being an information 
provider may have been the best option. 

Incident : 

(11) Mr. Zacc walks by and looks at the loan application form students are filling out. He says, "to fill 
this out right, you have to have something up here" [points] "because this is so long, you need to do 
[unintelligible]" 

Interpretation : 

Here Mr. Zacc is definitely an information provider — almost an information dictator. The kids 
didn't ask for his help; they seemed to be doing their work fine but he came over and told them 
what to do next and how to do certain things. 

Incident : 

(1 1) A student asks, "What do you put for bank?" Mr. Z points out how to read the credit report. One 
item is a credit union. Students ask what this is. Mr. Z says it like an employees bank. Mr. Z says they 
have to check everything out to see if the credit is good — “Is there a history of bouncing checks? Does 
the person have savings?" Then he has to show students again how to read the report. They are trying 
to use Chelsea Credit Union as a bank name (its ABC Credit Union, but they couldn't read it all, so 
rather than scroll, they assume Chelsea is also the name of the credit union) 

Interpretation : 

In this case Mr. Zacc is again providing all the information. He teaches them how to read the credit 
report and gives them the answers for the sheet in the process. 

Incident : 

(11) The students are reading the screen as the teacher walks past. They turn around as he goes by and 
ask "Is this what we need to do for you? Get one of these?" Mr. Zacc says they have to find the 
customer service rule that applies to today's lesson. 

Interpretation : 

Mr. Zacc's actions were desirable. He was just looking aroimd to see what the groups were doing. 
However, because there is a strong case of student dependance on the teacher, the students feel 
compelled to ask questions that, again, they should know the answers to fi'om past examples. 

Lessons Learned . The teachers at all four schools seem to want Chelsea Bank to be 
meaningful to the students. Yet it is clear that they cannot yet facilitate a learning environment of 
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this kind. There is little student-teacher interaction at all during these scenarios. When there is 
interaction, it is typically student-initiated and generally involves a question that the student 
could have answered for him or herself with some guidance. The dominant teacher responses to 
the students when they ask questions are directing responses. Often the teacher demands that the 
student go to the next step in a specified procedure. There is no discussion of why a certain step 
should come next or what the overall goal for the procedure is. The teachers do not ask the 
students what they had in mind for solving the problem. 

One teacher showed a significant amount of change over time. She moved to a point of 
being learner centered, promoting inquiry and critical thinking. Other than that, there was little 
growth across the three scenarios. The overall message here is that teachers can learn to be 
guides. They can be successfiil at asking questions that promote thinking and promoting students 
to think for themselves. However, they do not do this on their own. Some sort of training needs 
to be provided to help them understand how to become a guide rather than an information giver. 

Finally, after watching and analyzing these tapes, it becomes apparent that it does not take 
much longer to promote inquiry than to provide information. Most of the student-teacher 
interactions are very short regardless of the approach used. It is obvious that the teachers need 
more support in adopting the “promoting inquiry” model, but once they become proficient at it, 
the level of discussion in the classes will rise with little extra effort. 
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VI. Four Key Issues that Cut Across the Analyses 

The various aspects of the year-one study revealed several issues that cut across all of the 
analyses that were conducted. Thus these issues were observed from different perspectives and 
appeared important in understanding the behaviors of students and teachers using the program. 
T his suggests that these issues will be important considerations in the development of 
assessments and teacher-support materials for the Chelsea program. 

Four issues are presented here in brief form, for much has been said about each of them 
throughout the report; 

Teacher behavior makes a difference in student behavior. 

This issue was very obvious both to the researchers analyzing the program from the 
perspective of the SCANS competencies and to those analyzing the tapes to rate teacher and 
student behavior on major emphases and as problem-solving strategies. When the teacher was 
present, the students were noticeably dependent on him or her for guidance. Sometimes the 
teacher asked questions that were intended to help the students become more independent 
problem solvers; many other times the questions focused on how the consideration of the 
scenario’s problem should proceed and be completed. Much teacher guidance tended to focus on 
how to use the thinking/review tools built into the program. 

While students’ focus on the problem scenario was more obvious when the teacher was 
present, their progress was slowed by a seeming need to seek approval from the teacher; and 
there tended to be less collaborative effort among them. 

Yet it was clear from the study of the videotapes, from interviews with the teacher, from 
discussion with them in case studies, and from other classroom observations — ^that the teachers 
using the progr am have faith in and a variety of expectations for the program. (See the last issue 
discussed here.) The teacher’s role, it seemed, varied as well, depending, in significant degree, on 
his or her teaching style and degree of enthusiasm for the activity. Thus the teacher appeared 
crucial in developing problem-solving behaviors, collaboration, and the basic skills applied in 
solving the Chelsea .fian/t problems. It appeared that it was the teacher, as well — supported by 
the administration — who would determine whether what is learned from the Chelsea Bank 
experience is exported to other classes and whether links are made across the curriculum. 

While solving realistic problems, students practice basic skills. 

As the students partake in the Chelsea Bank activity, they are always using and 
developing their basic skills. They are constantly reading from the screen — ^both to each other 
and in order to synthesize for and with the members of the team. They write numerous responses 
and conclusions onto the screen as prompted by the program, and as they do this, they invariably 
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act as each other’s editor, paying heed to their language usage — discussing diction, phrasing, and 
mechanics, including spelling, capitalization, and punctuation. 

Not only were the students communicating with the computer, but they were clearly 
learning how to listen to and relate their ideas and opinions to each other. The interpersonal skills 
necessary to explain and to persuade, and to take an occasional leadership role were often in play 
in the collaboration that the program promotes. 

In many of the scenarios, students are using their math skills as well, for the problems 
presented rely on and refer to numbers. This was an objective frequently cited by teachers. 

The students who use the Chelsea program appear to become more adept at using 
the computer as their experience within the scenarios progresses. They appeared impressively 
comfortable reading the monitor and using the mouse and keyboard. This basic technological 
skill was directly connected to using the reference/information tools and thinking guides built 
into the program, and these in turn relate to basic banking concepts that the students appear to 
learn easily. 

A key problem-solving strategy that is developed by the program is getting and 
maintainin g a focus on a problem. The problems are depicted by the scenario presented on the 
computer, and they usually maintain the focus necessary to move toward a solution. The students 
appear to learn the procedures for understanding and acting within the scenarios quite readily. 
They learn to use the tools and resources provided within the program. This ability to focus is not 
nearly all there is to problem solving according to most definitions, however, and the results of 
the study indicate that while the students use some of the eight problem-solving strategies 
identified for observation, they rarely use some that could be effective. Also, there is not much 
creativity in their solution seeking and problem solving. Overall, however, it appears that the 
Chelsea experience can help students learn to focus on and deal with problems that they 
encounter in the real world. 

The students using the Chelsea Bank program were able to work together. 

Across the videotapes — and especially in their teachers’ absence — ^the students using the 
program appear clearly able to collaborate while deciding on solutions to the problems presented 
to them at Chelsea Bank. Much of the collaboration comes about through what might be called 
“mutual verbalization/visualization.” They sometimes review briefly what they know about the 
problem, the customer involved, and the options presented for a solution. 

Students read together from the computer monitor, supportively correcting any 
misreadings; they assist each other in true team fashion while entering responses into the 
progr am, adding words, revising diction, and nodding in approval or shaking their heads in 
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disapproval. Seldom do teams discuss input before the entering of it begins, but there is more in- 
process revision than most researchers have noticed in student expression, in general. 

Much of the best discussion seems to occur at the point in dealing with the scenario 
problem where the students are entering the consequences. This generates discussion and 
thinking, although it does not seem structured to promote revision in the decision already 
selected. As one boy reminded his teammates: They had already approved a loan, so it was not to 
be affected by what they decided the consequences were. This part of the program seems 
somewhat unrealistic to the students; it appears a bit inane to some to think about the 
consequences in the standard format used to generate their response. Because of this, it is not 
clear whether the kind of collaboration generated would transfer and encourage collaboration in 
other parts of the students’ school day. 

The focus of the program is different for different teachers and students. 

It is of significant interest that while most participants have very positive reactions to and 
expectations about the Chelsea program, there were a myriad of objectives and goals 
expressed; and few of those were clear-cut across the teachers and students we watched and 
interviewed. 

The program effect most often noted by teachers was that it builds a tendency toward 
collaboration and effective teamwork. This is ironic, perhaps, since teacher presence did not 
often promote collaboration. Students, too, noted that the experience requires teamwork — ^but not 
always as a means of endorsing it. Still, the student who is regretting the need to collaborate with 
two fellow students is indeed being encouraged to do that, and most teachers feel it is a behavior 
truly worth developing. 

Some teachers felt that the program’s goal should be expressed in terms of its content — 
the math applied in understanding and dealing with the problems and/or the banking concepts 
learned. Several teachers expressed this by advising that assessment be in the form of a test to 
cover details that could be learned. Still others, were more interested in the processing of 
problems or, like some of their students, the application of basic skills. These teachers seemed to 
be acknowledging the need for a performance assessment. 

As with the appreciation of the collaboration required, many teachers also noted the value 
of developing their students as problem solvers — some of these with a distinct appreciation for 
the students’ experience in the particular (banking) arena, others eager to develop the process. 

Overall, however, there was no clearly articulated set of goals or objectives for the 
program — an observation that depicts quite clearly one of the objectives that can be set for the 
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assessment component that will be developed; It must help teachers and students understand and 
target the goals of the program. 
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